Skip to content

What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation

🕒 Published (v1): 2026-07-14 03:26 UTC · Source: Arxiv · link

Why this paper was selected

Decomposes what temporal benchmark scores actually measure in video VLMs; channel-use analysis; methodological contribution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Aggregate temporal benchmark scores for video VLMs conflate two distinct questions: whether a task requires multi-frame order at all ("task question"), and which input channel—pixel visual sequence, in-context timestamps, or RoPE positional encoding—the model uses to resolve that order ("channel question"). The authors introduce a label-free reversal-drop diagnostic that reverses pixel order while holding RoPE forward, revealing that Molmo2 is position-dominant (reads order from RoPE) while Qwen3-VL is visual-sequence-dominant (reads from pixels), even when both models score similarly on standard benchmarks. Activation patching confirms this channel split is a genuine internal architectural property.

Problem

Standard temporal video benchmark evaluation uses two diagnostics—frame-shuffle sensitivity and multi-frame gain—that answer only the task question (does a question need multiple frames?) and are blind to the channel question (when multiple frames are needed, does the model read temporal order from pixel motion or from positional encoding?). Two models with similar aggregate accuracy can resolve order by entirely opposite mechanisms, making their failure modes non-interchangeable in ways the score cannot surface.

Method

The protocol manipulates three temporal input channels independently and in conflict across three video VLM families (Molmo2 4B/8B/O-7B, Qwen3-VL 4B/8B, PerceptionLM 1B/3B) on TempCompass (1580 MCQs) and TVBench (2205 MCQs):

  1. Information ladder: Evaluates each question at blank-pixel, single-frame, and full-video conditions to bound the task question. Multi-frame gain = \(\text{acc}(\text{full}) - \text{acc}(\text{single frame})\); order-gain = \(\text{acc}(\text{full}) - \text{acc}(\text{shuffled})\).

  2. Channel conflict (rev-corr): Frames are presented in reverse order, but RoPE position indices are reset to the original forward ordering. Pixels indicate one temporal direction; positional encoding indicates the opposite. A position-dominant model answers the forward event (accuracy preserved); a visual-sequence-dominant model answers the reversed event (accuracy collapses).

  3. Reversal-drop (label-free): On benchmarks without paired reverse labels (TVBench), the channel signal is read as \(\text{acc}(\text{base}) - \text{acc}(\text{rev-corr})\). Tasks with deterministically reversed ground truth (e.g., moving direction) provide a paired check for free.

  4. Activation patching: Residual-stream activations at vision token positions from a base (unshuffled) forward pass are installed at LM layers (layer 0, early half, late half, all, or DeepStack injection depth layers 0–2) during a shuffled/reversed pass. Recovery boundary localizes which architecture routes corrected order content and where.

Key Contributions

  • Benchmark audit framework: Decomposes a temporal score into task-question and channel-question components; shows that shuffle sensitivity and multi-frame gain address only the former.
  • Reversal-drop diagnostic: A label-free screen for the channel question; drops onto any compatible temporal benchmark without new annotations; paired reverse labels enable exact reversal accuracy when available (TempCompass attribute change, direction).
  • Position-dominant vs. visual-sequence-dominant taxonomy: Empirical dissociation of Molmo2 (RoPE-dominant, bidirectional vision attention) vs. Qwen3-VL (pixel-dominant, 3D MRoPE with causal attention) at two scales and across two benchmarks.
  • Mechanistic validation via activation patching: Identifies architecture-specific recovery boundaries—first LM layer suffices for Molmo2 and PerceptionLM; Qwen3-VL's DeepStack mid-layer ViT injection creates a residual that cannot be fully corrected, confirming the behavioral split is internal.

Results

  • Aggregate similarity, channel divergence: Both Molmo and Qwen achieve overall multi-frame gain \(\approx +0.20\) and order-gain \(\approx +0.17\) on TempCompass (standard diagnostics rate them equal).
  • TVBench reversal-drop (Table 3):
  • Molmo2: near-invariant to pixel reversal across all temporal tasks (mean drop \(+0.04\); largest single task \(+0.14\) at 8B on egocentric sequence).
  • Qwen3-VL: collapses on every temporal task (drops \(+0.24\)–\(+0.55\) on scene transition, moving direction, action sequence, action localization, egocentric sequence at both 4B and 8B).
  • TempCompass paired ground truth (Table 4):
  • Under rev-corr, Molmo2 shows near-total RoPE override on attribute change: \(F - E \approx -0.95\) at 4B/8B, \(-0.82\) at O-7B.
  • Qwen3-VL shows the same direction at far smaller magnitude, weakening with scale: \(-0.23\) (4B), \(-0.14\) (8B).
  • Motion-dependent slice: Molmo2's RoPE override holds or strengthens (\(F - E \approx -1.0\)) on items not answerable from a single frame; Qwen3-VL's override vanishes on motion-dependent items at both scales.
  • Activation patching: Installing corrected activations at layer 0 alone fully recovers Molmo2 and PerceptionLM; Qwen3-VL leaves a residual attributable to DeepStack injection at LM layers 0–2.

Limitations

  • Conflict-based intervention requires access to model internals (RoPE manipulation); restricted to open-weight models.
  • The reversal-drop is relabel-free on TVBench but cannot distinguish "disrupted by conflict" from "reads pixels" without paired ground truth—TempCompass's paired videos are needed for exact quantification.
  • Only three model families tested; generalization to other architectural choices (e.g., different positional encoding schemes or fusion strategies) is not demonstrated.
  • The timestamp channel is studied descriptively as a model-level partition, not as a calibrated reliance estimate.
  • DeepStack ablations are limited to Qwen3-VL; the analysis of whether other mid-layer injection schemes produce the same residual is left open.

Relevance to Vision-Language Models

This work directly challenges how the video VLM community validates temporal understanding, showing that standard diagnostic scores are insufficient to distinguish models that encode time via visual motion versus positional indexing. For researchers building or benchmarking video VLMs, the reversal-drop protocol provides a plug-in diagnostic that exposes a failure mode invisible to current evaluations: a position-dominant model will fail whenever positional encodings are unreliable or absent, while a visual-sequence-dominant model will fail when visual ordering is corrupted. The mechanistic account via activation patching further connects the behavioral split to concrete architectural choices (bidirectional vs. causal attention over visual tokens, 1D vs. 3D MRoPE, DeepStack injection depth), giving VLM designers specific levers to tune. This frames a broader lesson for the VLM benchmarking literature: aggregate accuracy on capability benchmarks can mask systematic internal route differences that produce opposite failure modes across deployment conditions.