Skip to content

VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; VideoReasonBench: vision-centric complex video reasoning benchmark

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoReasonBench is a benchmark of 1,440 questions over 240 videos designed to evaluate vision-centric, complex video reasoning in MLLMs. It requires models to track dense sequences of operations applied to partially-observable latent states, organized into three escalating skill levels: recall, inference, and counterfactual prediction. Most state-of-the-art MLLMs fail badly (GPT-4o: 6.9%), while Gemini-2.5-Pro achieves 56.0% vs. 73.8% human accuracy.

Problem

Existing video benchmarks (Video-MME, TempCompass, MMVU) are answerable without activating extended chain-of-thought reasoning—Gemini-2.5-Flash answers them correctly with near-zero thinking tokens. Recent reasoning-oriented video benchmarks (VCR-Bench, MINERVA) still rely heavily on short-horizon temporal skills (action counting, temporal order) rather than deep visual reasoning. No benchmark reliably demonstrates that extended CoT thinking benefits video understanding, nor quantifies the dependence of tasks on actual visual content.

Method

Each video is formalized as a sequence of state transitions \(\{S_t, o_t, S_{t+1}\}_{t=1}^{T-1}\), where all operations \(o_t\) are visually observable but the states \(S_t\) are only partially revealed (either the initial or final state is shown). Six video categories are constructed: - Number: sliding-tile puzzle on an \(N \times N\) board - Circle: piece-flipping on a grid via a moving red circle - Cup: coin-hiding with cup swaps - File: file-system create/delete/copy/move operations - Card: stack operations (add top / remove bottom) - Chip: chip counting in cups

Videos use \(T \in [5, 14]\) operations and \(N \in \{3, 4\}\) board sizes, with synthetic rendering (Matplotlib), command-line screenshots, or manual real-world recordings. A semi-automatic pipeline (video engine + question engine) generates question–answer pairs via templated prompts and hand-crafted rule-based answer derivation. Six reasoning skills span three levels (two per level): Recall Order, Recall Count (L1); Infer State, Compare State (L2); Predict State, Predict Operation (L3). Evaluation uses Qwen2.5-72B as an LLM judge; for Predict Operation (no unique ground truth), extracted operations are simulated and the resulting state is verified programmatically.

Key Contributions

  • Formal definition of vision-centric complex video reasoning as latent-state inference over observable operation sequences
  • 1,440-question benchmark with 240 videos across 6 demonstration types and 6 reasoning skills at 3 progressive levels
  • Comprehensive evaluation of 18 MLLMs revealing near-total failure of non-thinking models (<10% accuracy for all non-Gemini-2.5 models) and a large gap to human performance (73.8%)
  • Empirical demonstration that extended thinking budget yields ~9% gain on VideoReasonBench but <2.5% on four existing video benchmarks, establishing it as a paragon for CoT-sensitive evaluation
  • Ablation showing VideoReasonBench is uniquely vision-dependent: cutting 50% of video frames drops accuracy by 55.5% (vs. ≤10.3% on other benchmarks); text-only drops 96.4% (vs. 27–50%)

Results

  • Gemini-2.5-Pro-0506 (thinking): 56.0% overall; best per level: Recall Order 69.2%, Recall Count 70.4%, Infer State 63.3%, Compare State 56.7%, Predict State 42.1%, Predict Operation 34.6%
  • Human baseline (240-example subset, avg 223.2s/question): 73.8%
  • Gemini-2.5-Flash (thinking): 27.4%; (no thinking): 18.8%
  • o4-mini (thinking): 10.7%; Seed1.5-VL (thinking): 11.5%
  • GPT-4o (no thinking): 6.9%; Gemini-2.0-Flash: 10.4%
  • Open-source flagship 72B models (LLaVA-OneVision-72B, LLaVA-Video-72B): 0.1%; Qwen2.5-VL-72B: 7.2%; InternVL3-78B: 5.8%
  • All efficient (<10B) models: <2%
  • vid2txt baseline (video replaced by textual summary): Gemini-2.5-Flash rises from 27.4% → 72.2%; Seed1.5-VL rises from 11.5% → 69.4%, isolating temporal visual perception as the primary bottleneck
  • Thinking budget sweep on Gemini-2.5-Flash: VideoReasonBench gains ~9% from budget 0→8192 tokens; TempCompass +1.0%, Video-MME +2.4%, MMVU +0.3%, Video-MMMU +1.9%

Limitations

  • Only 240 videos (60 per category); relatively small scale may limit coverage of real-world video diversity
  • Six categories are all structured, rule-governed domains (puzzles, file systems, card stacks)—may not generalize to open-world naturalistic videos
  • Text-only and vid2txt baselines show fine-grained visual perception is the dominant bottleneck, but the benchmark cannot disentangle perceptual limits from tokenization/sampling artifacts in video processing
  • Human accuracy ceiling (73.8%) is itself below 100% due to cognitive load of tracking up to 14 sequential operations, suggesting the benchmark is hard even for humans, which complicates calibration

Relevance to Vision-Language Models

VideoReasonBench directly exposes a critical failure mode of current VLMs: the inability to faithfully perceive and track dense sequential visual operations, even when the reasoning capacity exists (as shown by the vid2txt vs. full-video gap of ~45% for Gemini-2.5-Flash). The benchmark provides a principled methodology for diagnosing whether performance improvements in MLLMs are driven by visual grounding or language priors, a key question for the VLM community. The finding that extended CoT thinking is necessary but insufficient—perception is the bottleneck—suggests future VLM work should prioritize high-fidelity temporal frame encoding and state-tracking mechanisms rather than solely scaling reasoning. It also serves as a calibration point for test-time compute scaling research applied to vision: unlike language tasks, video reasoning gains from thinking are real but contingent on sufficient visual input quality.