Skip to content

O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning

🕒 Published (v1): 2026-07-20 16:36 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; object-centric VLM reasoning for industrial anomaly detection

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

O-VAD is a training-free agentic framework for industrial video anomaly detection (IVAD) that replaces direct VLM prompting with a three-stage pipeline: object grounding via VLM + SAM3, per-object spatiotemporal state tracking via CropFormer + SAM2, and cascaded chain-of-thought (CoT) anomaly reasoning over accumulated state trajectories. It requires no domain-specific labels, predefined anomaly taxonomy, or fine-tuning, and outputs structured reports with anomaly type, severity, affected object, temporal localization, and causal explanation.

Problem

Existing VLM-based anomaly detection either (1) injects external knowledge or normal samples as context, causing responses that prioritize contextual plausibility over detection accuracy, or (2) operates at the frame/video level, losing the object-centric physical dynamics that industrial inspection requires. Traditional VAD methods follow a "one-class-one-model" paradigm requiring per-class normal training data and emit only binary scores with no semantic rationale. IVAD is specifically harder due to complex object transformations (pressing, cutting, assembly), strict physics/procedural constraints, and the need for interpretable root-cause explanations.

Method

Stage 1 — Automated Object Grounding. \(N_f\) sampled frames are sent to a VLM (GPT-5) to produce a deduplicated object inventory \(\mathcal{O} = \{(n_k, d_k, b_k)\}\); each object is then segmented via SAM3 to yield initial masks \(\mathcal{M}_1\).

Stage 2 — Object-Centric State Tracking. CropFormer produces per-frame spatial partitions; SAM2 propagates them into spatiotemporal tubelets. Missing tracks caused by transformations (breakage, material release) are recovered by combining: $\(S_{\text{prox}}(C,P)=\max_{j\in\{1,2,3\}}\frac{|c_s \cap m_s^j|}{|c_s|}, \quad S_{\text{sem}}(C,P)=\max_{i<s,\,j\geq s}\; f(p_i,I_i)\cdot f(c_j,I_j)^\top\)$ where \(f(M,I)\) is mask-pooled CLIP embedding. At key temporal transitions the VLM is prompted with masked frame pairs to produce open-ended state-change events \(e=(t_{\text{start}}, t_{\text{end}}, \textit{type}, \textit{cause}, \textit{desc}, \textit{sev}, k)\).

Stage 3 — State-Aware Anomaly Reasoning. A 6-step cascaded CoT (process understanding → observation → expectation → comparison → causation → classification/severity) operates over accumulated state trajectories. Anomaly confidence: $\(\hat{y}_{t,k} = \theta_{\text{VLM}}(p_{\text{CoT}} \oplus \mathcal{S}_k \oplus c), \quad \hat{y}^{\text{frm}}_t = \max_k \hat{y}_{t,k}, \quad \hat{y}^{\text{vid}} = \max_t \hat{y}^{\text{frm}}_t\)$ Candidate anomalies with \(c_{\text{orig}} \in (\tau_{\text{lo}}, \tau_{\text{hi}}) = (0.2, 0.8)\) are gated by a visual verification call; the final confidence is \(c_{\text{final}}(a) = c_{\text{orig}}(a) \cdot c_{\text{ver}}\) (or \(\cdot (1-c_{\text{ver}})\) if unverified); anomaly retained if \(c_{\text{final}} \geq \tau_{\text{conf}} = 0.3\).

Key Contributions

  • Training-free "ground→track→reason" agentic pipeline requiring no domain labels, predefined taxonomy, or fine-tuning.
  • Object-centric state tracker with open-ended, free-form state-change annotation, enabling detection of novel phenomena not covered by fixed taxonomies.
  • Post-visual verification with confidence-tiered gating to suppress false positives without an extra model.
  • Data curation pipeline producing fine-grained object trajectory annotations for three IVAD benchmarks (Phys-AD, LiquidAD, IPAD).
  • Interpretable structured anomaly reports (type, severity, object ID, frame range, causal chain) rather than binary scores.

Results

All comparisons from Table 1; "†" = trained, "∗" = training-free. Baselines: MNAD.p†, S3R† (traditional VAD); Qwen3-VL-32B∗, GPT-5∗ (direct prompting); URF∗, VERA† (agentic workflows).

  • Phys-AD video-level: O-VAD achieves best AUROC (0.584) among all methods; F1 = 0.671 vs GPT-5 = 0.630 and Qwen3 = 0.486 (both training-free); trained MNAD.p leads on F1 (0.755). Type-level LLM-judge: O-VAD = 0.595 vs GPT-5 = 0.580 and Qwen3 = 0.372.
  • LiquidAD video-level: O-VAD F1 = 0.929, AUROC = 0.692 (best AUROC); URF achieves slightly higher F1 (0.949) but lower AUROC (0.365). Frame-level F1 = 0.507, AUROC = 0.512 vs S3R (best among trained: F1 = 0.679, AUROC = 0.625).
  • IPAD video-level: O-VAD R = 0.954 (best), F1 = 0.714 (close to URF = 0.721), AUROC = 0.565 (best). Frame-level F1 = 0.338, AUROC = 0.518.
  • O-VAD is the only method producing semantic anomaly-type outputs (BERTScore, LLM-judge) while simultaneously being competitive with trained methods on discriminative metrics.

Limitations

  • Depends on proprietary frontier models (GPT-5 for VLM reasoning, SAM3 for segmentation), making cost and reproducibility dependent on API availability.
  • Frame-level detection performance remains modest on IPAD (F1 = 0.338, AUROC = 0.518), suggesting temporal localization precision is still below trained discriminative methods.
  • IPAD anomalies are defined as deviations from reference normal samples rather than commonsense physics, making the purely training-free commonsense approach structurally less suited to that dataset's protocol.
  • URF (another training-free agentic workflow) outperforms O-VAD on LiquidAD video-level F1 (0.949 vs 0.929), indicating O-VAD is not universally dominant across all metrics.
  • No ablation on the sensitivity of track-recovery thresholds (\(\tau_{\text{prox}}, \tau_{\text{sem}}\)) is visible in the provided text.

Relevance to Vision-Language Models

O-VAD is a direct study in how to elicit fine-grained spatiotemporal reasoning from VLMs without fine-tuning: rather than feeding raw video and hoping for zero-shot detection, it structures the VLM's input as per-object state-trajectory evidence, converting a hard visual-reasoning problem into a structured CoT task. This "evidence-first, reasoning-second" design is a meaningful departure from prevailing direct-prompting paradigms and demonstrates that the failure mode of frontier VLMs on specialized perception tasks can often be attributed to impoverished input structure rather than to model capacity. For VLM researchers, the work also benchmarks GPT-5 and Qwen3-VL-32B in an extended-thinking zero-shot regime on a temporally dense industrial domain, exposing their current limits in object-centric physical reasoning and providing a concrete framework for how auxiliary tracking modules (SAM2, CropFormer, CLIP) can compensate for VLM blind spots.