Skip to content

From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA

🕒 Published (v1): 2026-06-29 12:33 UTC · Source: Arxiv · link

Why this paper was selected

Akata group; fundamental audit of whether VLMs truly use visual evidence

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Recent open-weight VLMs on traffic accident VideoQA benchmarks often answer correctly without any video input, and on MM-AU adding more frames degrades accuracy. The authors introduce two dataset-level diagnostics (Blind Gap, Visual Gain) and a continuous per-question Shortcut Score to detect and filter shortcut-prone instances, producing evaluation subsets with measurably stronger visual grounding.

Problem

Benchmark accuracy alone cannot verify that a model actually uses visual evidence. In traffic accident VideoQA—a safety-critical domain where answers should depend on scene-specific visual cues—models may exploit textual shortcuts (repetitive answer patterns, language priors) instead of performing genuine visual reasoning. Prior work either applies binary filtering (discarding questions answered correctly without video) or provides only dataset-level metrics; neither identifies which individual questions are shortcut-prone.

Method

  1. Blind Gap (BG) \(= \text{Acc}_\text{blind} - \text{Acc}_\text{random}\) — measures text-only accuracy above chance at the dataset level.
  2. Visual Gain (VG) \(= \text{Acc}_\text{video} - \text{Acc}_\text{blind}\) — measures the marginal benefit of providing video input.
  3. Shortcut Score \(S(q) = T(q) - V(q)\), where:
  4. \(T(q)\) is the Textual Evidence Score: average confidence-weighted correctness across a panel of text-only LLMs (Llama 3.1 8B, Mistral 7B, Qwen 2.5 14B).
  5. \(V(q)\) is the Visual Necessity Score: average normalized probability gain on the correct answer when video is added, computed across four VLMs (InternVL2.5/3.5 8B, Qwen2.5-VL 7B, LLaVA-OV 8B).
  6. Filtering: retain questions with \(S(q) \le \tau\) (set \(\tau=0.1\) after threshold sweep). This is training-free and continuous, unlike prior binary-filter baselines (LLM-Inspector, B-Clean).

Key Contributions

  • First systematic audit of modality collapse in traffic accident VideoQA across four public benchmarks (MM-AU, VRU-Accident, SUTD-TrafficQA, AccidentBench).
  • Blind Gap and Visual Gain — lightweight dataset-level diagnostics that jointly characterize shortcut exploitation vs. genuine visual grounding.
  • Shortcut Score — a continuous, training-free, instance-level metric combining confidence-weighted text-only evidence with visual necessity. Enables principled filtering without discarding visually-grounded-but-blind-correct questions.
  • Empirical demonstration that S-score filtering yields grounded subsets where text-only models fall to near-random and VG turns positive, outperforming binary baselines that collapse VG to negative values.

Results

  • MM-AU is the worst offender: average BG = 33.25%, VG = –4.62% (video hurts). Text-only LLMs match or exceed VLMs. Increasing frames (8 → 64 → 256) monotonically decreases accuracy.
  • TrafficQA is the most visually grounded: average BG = 5.89%, average VG = 11.83%.
  • VRU-Accident shows a mixed profile: high BG (21.91%) but also high positive VG (15.70%) — visual signal exists alongside strong language priors.
  • S-score filtering (LLaVA-OV held out):
  • MM-AU: BG drops from 25.70 → 10.38; VG flips from –0.24 → +7.42 (836/2453 questions retained).
  • VRU-Accident: BG roughly halved (25.10 → 18.76); VG nearly doubles (12.93 → 25.49).
  • Binary baselines degrade VG: B-Clean yields VG = –19.91 on MM-AU and –10.65 on VRU-Accident (removes visually useful questions alongside shortcuts).
  • MM-AU's shortcut susceptibility is linked to repetitive answer phrasing: one exact answer template appears 234 times across the dataset.

Limitations

  • Scale: all VLMs evaluated at 7–8B parameters; results may not generalize to larger models.
  • Frame sampling: fixed 16-frame uniform sampling may undersample long videos (AccidentBench videos up to ~16 min) or miss fine-grained temporal cues.
  • Shortcut Score calibration depends on probability estimates that may be miscalibrated in open-weight VLMs; the threshold \(\tau\) is a free parameter needing per-benchmark tuning.
  • Format constraint: all benchmarks are multiple-choice; diagnostic metrics may not transfer to open-ended VideoQA.
  • Limited baselines: only three text-only LLMs used for \(T(q)\); broader model panels could change per-question scores.
  • Not a training intervention: filtering improves evaluation integrity but does not mitigate modality collapse in model training.

Relevance to Vision-Language Models

This work directly addresses modality collapse — the failure mode where VLMs' strong language priors dominate visual input, a known issue in the broader VLM literature (Sim et al., 2025; Deng et al., 2025; Ghosh et al., 2026). For researchers tracking VLMs, the paper provides practical, lightweight diagnostics to audit whether benchmark gains reflect genuine visual grounding or shortcut exploitation, and proposes a filtering method to construct more trustworthy evaluation subsets. The finding that MM-AU is entirely solvable without video—and that adding frames makes things worse—is a strong cautionary signal for safety-critical VLM deployment.