Skip to content

O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning

🕒 Published (v1): 2026-07-20 16:36 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

O-VAD is a training-free agentic framework for industrial video anomaly detection (IVAD) that replaces direct VLM prompting with a three-stage pipeline: object grounding via VLM + SAM3, per-object spatiotemporal state tracking via CropFormer + SAM2, and cascaded chain-of-thought (CoT) anomaly reasoning over accumulated state trajectories. It requires no domain-specific labels, predefined anomaly taxonomy, or fine-tuning, and outputs structured reports with anomaly type, severity, affected object, temporal localization, and causal explanation.

Problem

Existing VLM-based anomaly detection either (1) injects external knowledge or normal samples as context, causing responses that prioritize contextual plausibility over detection accuracy, or (2) operates at the frame/video level, losing the object-centric physical dynamics that industrial inspection requires. Traditional VAD methods follow a "one-class-one-model" paradigm requiring per-class normal training data and emit only binary scores with no semantic rationale. IVAD is specifically harder due to complex object transformations (pressing, cutting, assembly), strict physics/procedural constraints, and the need for interpretable root-cause explanations.

Method

Stage 1 — Automated Object Grounding. \(N_f\) sampled frames are sent to a VLM (GPT-5) to produce a deduplicated object inventory \(\mathcal{O} = \{(n_k, d_k, b_k)\}\); each object is then segmented via SAM3 to yield initial masks \(\mathcal{M}_1\).

Stage 2 — Object-Centric State Tracking. CropFormer produces per-frame spatial partitions; SAM2 propagates them into spatiotemporal tubelets. Missing tracks caused by transformations (breakage, material release) are recovered by combining: $\(S_{\text{prox}}(C,P)=\max_{j\in\{1,2,3\}}\frac{|c_s \cap m_s^j|}{|c_s|}, \quad S_{\text{sem}}(C,P)=\max_{i<s,\,j\geq s}\; f(p_i,I_i)\cdot f(c_j,I_j)^\top\)$ where \(f(M,I)\) is mask-pooled CLIP embedding. At key temporal transitions the VLM is prompted with masked frame pairs to produce open-ended state-change events \(e=(t_{\text{start}}, t_{\text{end}}, \textit{type}, \textit{cause}, \textit{desc}, \textit{sev}, k)\).

Stage 3 — State-Aware Anomaly Reasoning. A 6-step cascaded CoT (process understanding → observation → expectation → comparison → causation → classification/severity) operates over accumulated state trajectories. Anomaly confidence: $\(\hat{y}_{t,k} = \theta_{\text{VLM}}(p_{\text{CoT}} \oplus \mathcal{S}_k \oplus c), \quad \hat{y}^{\text{frm}}_t = \max_k \hat{y}_{t,k}, \quad \hat{y}^{\text{vid}} = \max_t \hat{y}^{\text{frm}}_t\)$ Candidate anomalies with \(c_{\text{orig}} \in (\tau_{\text{lo}}, \tau_{\text{hi}}) = (0.2, 0.8)\) are gated by a visual verification call; the final confidence is \(c_{\text{final}}(a) = c_{\text{orig}}(a) \cdot c_{\text{ver}}\) (or \(\cdot (1-c_{\text{ver}})\) if unverified); anomaly retained if \(c_{\text{final}} \geq \tau_{\text{conf}} = 0.3\).

Key Contributions

  • Training-free "ground→track→reason" agentic pipeline requiring no domain labels, predefined taxonomy, or fine-tuning.
  • Object-centric state tracker with open-ended, free-form state-change annotation, enabling detection of novel phenomena not covered by fixed taxonomies.
  • Post-visual verification with confidence-tiered gating to suppress false positives without an extra model.
  • Data curation pipeline producing fine-grained object trajectory annotations for three IVAD benchmarks (Phys-AD, LiquidAD, IPAD).
  • Interpretable structured anomaly reports (type, severity, object ID, frame range, causal chain) rather than binary scores.

Results

All comparisons from Table 1; "†" = trained, "∗" = training-free. Baselines: MNAD.p†, S3R† (traditional VAD); Qwen3-VL-32B∗, GPT-5∗ (direct prompting); URF∗, VERA† (agentic workflows).

  • Phys-AD video-level: O-VAD achieves best AUROC (0.584) among all methods; F1 = 0.671 vs GPT-5 = 0.630 and Qwen3 = 0.486 (both training-free); trained MNAD.p leads on F1 (0.755). Type-level LLM-judge: O-VAD = 0.595 vs GPT-5 = 0.580 and Qwen3 = 0.372.
  • LiquidAD video-level: O-VAD F1 = 0.929, AUROC = 0.692 (best AUROC); URF achieves slightly higher F1 (0.949) but lower AUROC (0.365). Frame-level F1 = 0.507, AUROC = 0.512 vs S3R (best among trained: F1 = 0.679, AUROC = 0.625).
  • IPAD video-level: O-VAD R = 0.954 (best), F1 = 0.714 (close to URF = 0.721), AUROC = 0.565 (best). Frame-level F1 = 0.338, AUROC = 0.518.
  • O-VAD is the only method producing semantic anomaly-type outputs (BERTScore, LLM-judge) while simultaneously being competitive with trained methods on discriminative metrics.

Limitations

  • Depends on proprietary frontier models (GPT-5 for VLM reasoning, SAM3 for segmentation), making cost and reproducibility dependent on API availability.
  • Frame-level detection performance remains modest on IPAD (F1 = 0.338, AUROC = 0.518), suggesting temporal localization precision is still below trained discriminative methods.
  • IPAD anomalies are defined as deviations from reference normal samples rather than commonsense physics, making the purely training-free commonsense approach structurally less suited to that dataset's protocol.
  • URF (another training-free agentic workflow) outperforms O-VAD on LiquidAD video-level F1 (0.949 vs 0.929), indicating O-VAD is not universally dominant across all metrics.
  • No ablation on the sensitivity of track-recovery thresholds (\(\tau_{\text{prox}}, \tau_{\text{sem}}\)) is visible in the provided text.

Relevance to Harnesses / Meta-Harnesses

O-VAD is a concrete harness instantiation: it orchestrates four specialist models (VLM, SAM3, CropFormer, SAM2) in a fixed three-stage pipeline where each stage's output is structured state that the next stage consumes, culminating in a CoT reasoning call over accumulated trajectories rather than raw video — exactly the pattern of "perception tools feeding a reasoning layer" that meta-harness designers care about. The key transferable lesson is that decomposing perception into object grounding → spatiotemporal tracking → causal reasoning yields far better structured outputs than monolithic VLM prompting, which validates the general harness principle of staging context accumulation before invoking expensive reasoning. For someone building agent harnesses, O-VAD also demonstrates a clean no-fine-tuning composition strategy: domain-specific capability is injected through tool selection and prompt design (the CoT schema), not model weights, making the architecture portable across industrial verticals.