Skip to content

Do LVLMs Truly Understand Video Anomalies? Revealing Hallucination via Co-Occurrence Patterns

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LVLMs deployed for Video Anomaly Detection (VAD) hallucinate anomalies by relying on statistical co-occurrence shortcuts between visual objects/interactions and anomaly-related text learned during pretraining, rather than performing genuine scene-aware reasoning. The authors systematically characterize this bias across four semantic levels (object, combination, interaction, temporal) and propose VAD-DPO, a Direct Preference Optimization framework using visually similar but semantically contrasting video pairs to suppress shortcut reliance. VAD-DPO achieves state-of-the-art AUC on six VAD benchmarks using only a 2B-parameter backbone, with an 8.3% gain on scene-dependent anomaly detection.

Problem

Existing LVLM-based VAD methods are not evaluated for whether their decisions are grounded in scene semantics or driven by spurious vision-text co-occurrence patterns. Objects like "fire," "gun," or "blood" co-occur overwhelmingly with anomaly-related text in pretraining corpora (Bias Scores of 5.24, 4.40+), causing models to misclassify semantically normal scenes containing these objects as anomalous (false positive rates of 67–79% across architectures and scales). This hallucination extends beyond isolated objects to combinations, interactions (e.g., "hitting"), and temporal motion patterns, but is underexplored and unaddressed by prior VAD work.

Method

Hallucination Diagnosis: Co-occurrence statistics are computed on LAION-CC-SBU (558K pairs) by detecting objects via YOLOv5 and interactions via GLIP, then counting their co-occurrence with GPT-generated anomaly/normality phrase sets. Two controlled probe sets (40 samples × 4 object categories = 200 total) test false positive rates (normal scenes with anomalous-looking objects inserted) and false negative rates (anomalous scenes with normal-looking objects).

VAD-DPO: Built on standard DPO (Rafailov et al. 2023), the method constructs contrastive preference pairs: video clips that are visually similar (same object/interaction present) but semantically contrasting in anomaly label (e.g., "person hitting another in a boxing ring" [normal] vs. "in a street alley" [anomalous]). Pairs are sourced from HIVAU-70K and supplemented with Wan 2.1-generated synthetic clips. The training objective combines the standard DPO contrastive loss with an anchored term:

\[L_{\text{total}} = L_{\text{VAD-DPO}} + \gamma \cdot L_{\text{Anc}}\]

where \(L_{\text{Anc}}\) penalizes divergence from the reference model only on the preferred (correct) response, stabilizing training. Base model: Qwen2-VL 2B; \(\gamma=1\); single epoch; lr=1e-4.

Inference: LAVAD-style protocol — 8 uniformly sampled frames scored with a 0–1 suspicion prompt, then temporal attention weights from the decoder propagate sparse scores to neighboring frames for dense per-frame anomaly score sequences.

Key Contributions

  • First systematic study of hallucination in LVLM-based VAD, characterizing co-occurrence bias at four levels: object, object-combination, interaction, and temporal dynamics.
  • Quantitative probe framework with Bias Score / Normalized Abnormal Frequency metrics and two diagnostic probe sets (false positive / false negative) covering 7 LVLMs across scales.
  • VAD-DPO: a preference optimization training recipe using counter-example pairs that explicitly targets shortcut suppression via a combined contrastive + anchored DPO loss.
  • Demonstrated that preference tuning on a small, carefully constructed dataset also improves general video understanding benchmarks (MVBench, PerceptionTest, EgoSchema, Video-MME).

Results

  • Detection AUC (Table 4): VAD-DPO (2B) outperforms all baselines including 7B–34B LVLMs on all five evaluated datasets:
  • ShanghaiTech: 87.2% (vs. Holmes-VAU 85.2%, best prior)
  • UCF-Crime: 86.2% (vs. Holmes-VAU 84.5%)
  • XD-Violence: 88.5% (vs. Holmes-VAU 86.3%)
  • NWPU Campus: 79.1% (vs. Holmes-VAU 70.8%; +8.3 pp)
  • MSAD: 85.4% (vs. Holmes-VAU 81.6%)
  • Shortcut probe (Table 5): VAD-DPO reduces average FPR from >75% (expert models) to 13.1% — an absolute reduction of >80%; FNR drops to 12.5% vs. 77.5–82.5% in prior methods.
  • Ablation (Table 6, NWPU Campus): Instruction tuning 69.2% → PPO 72.3% → DPO 73.7% → VAD-DPO 79.1%.
  • Generalization (Table 7): Consistent ~0.4–0.8 pp improvement over base Qwen2-VL-2B on MVBench, PerceptionTest, EgoSchema, Video-MME.

Limitations

  • The co-occurrence analysis is conducted solely on LAION-CC-SBU (558K pairs), which is only part of the pretraining corpus for LLaVA; the bias magnitudes may not generalize to other LVLMs with different pretraining corpora.
  • Preference pair construction relies on existing VAD datasets plus synthetic video generation (Wan 2.1); coverage of edge-case co-occurrence patterns is incomplete and not exhaustively characterized.
  • Temporal co-occurrence analysis is limited to one scenario (person standing still on a busy road); the interaction between temporal dynamics and shortcut suppression is incompletely explored.
  • The method is demonstrated only on Qwen2-VL (2B); it is unclear how VAD-DPO scales to larger LVLMs or whether the same \(\gamma\) hyperparameter generalizes.
  • Future directions (GRPO-style optimization) are noted but not experimentally validated.

Relevance to Vision-Language Models

This paper directly probes a fundamental reliability question for VLMs deployed in safety-critical settings: do models reason from visual evidence or from pretraining co-occurrence statistics? The co-occurrence hallucination phenomenon identified here is a domain-specific instance of the broader object hallucination problem in VLMs, but uniquely studied in a temporal/video context and across multiple semantic abstraction levels. VAD-DPO contributes a concrete DPO recipe for suppressing spurious visual-textual correlations without sacrificing general video understanding, which is broadly applicable to any VLM fine-tuning scenario involving context-dependent label assignment. The multi-level diagnostic framework (object → combination → interaction → temporal) offers a reusable methodology for auditing shortcut reliance in any VLM application domain.