Skip to content

IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Image-grounded video benchmark tests VLMs on cross-modal video comprehension

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IV-Bench is the first manually annotated benchmark for evaluating MLLMs on image-grounded video perception and reasoning, where each query pairs an externally sourced reference image with a video—both are required to answer correctly. Across 28 evaluated models, the best achieves only 28.9% accuracy versus 88.8% for humans, exposing a large capability gap. The benchmark establishes that scaling model size yields only modest gains, particularly on reasoning-intensive tasks.

Problem

Existing video benchmarks use text-only queries, which cannot precisely specify fine-grained visual attributes (e.g., a specific person's face, subtle object variants). Prior image-text video benchmarks (V2P-Bench, VideoRefer-Bench) extract query images directly from the video, reducing diversity and enabling frame-matching shortcuts. Video-MMMU uses external images but treats video as auxiliary—video is not required to answer. No benchmark evaluates the scenario where both an external reference image and the video are jointly necessary for answering, leaving image-grounded video perception and reasoning systematically unmeasured.

Method

IV-Bench collects 966 videos (each ≥5 min) across five categories (Knowledge, Film & Television, Sports, Artistic Performance, Life Record) and annotates 2,560 image-text queries spanning 13 tasks: 7 perception tasks (Existence, Reverse Existence, Natural Language Inference, Spatial Relationship, Keyframe Extraction, Constrained OCR, Detailed Events) and 6 reasoning tasks (Counting, Space-Time Computing, Summary, Instruction Understanding, Attribute Change, Temporal Reasoning). Query images are sourced externally (not frame-extracted), preventing perceptual similarity shortcuts. A two-round quality control process is applied: the first round checks clarity, correctness, and category labels; the second removes samples solvable without the image or video, and enforces at least two "effective" distractors—answers that would be correct if a different image were provided, thereby certifying image necessity. Evaluation uses 32 uniformly sampled frames with a video-first input ordering (frames → image → question).

Key Contributions

  • IV-Bench: first comprehensive benchmark requiring both an external reference image and video to answer each query, with 966 videos and 2,560 annotated image-text queries across 13 tasks and 5 categories.
  • Systematic evaluation of 28 MLLMs (23 open-source + 5 closed-source), revealing best model accuracy of 28.9% vs. 88.8% human accuracy.
  • Ablation analysis showing video-first input ordering outperforms image-first for larger models; smaller models fail to leverage image context regardless of ordering.
  • Demonstration that visual token budget is better spent on more frames than higher resolution.

Results

  • Best overall model: Qwen2.5-VL-72B at 28.9%; InternVL2.5-78B at 28.6%; Gemini-2.0-Pro at 27.7%.
  • Human accuracy: 88.8% overall; 91.5% on perception, 86.9% on reasoning.
  • Best task for models: NLI (Qwen2.5-VL-72B: 64.7%); hardest task: Temporal Reasoning (top model: 16.7%).
  • Models under 10B: best is Qwen2.5-VL-7B at 18.5% (random baseline ≈ 11.1%).
  • Scaling gains are perception-skewed: Qwen2.5-VL-72B vs. 7B gains +30.8% on Existence and +43.1% on NLI, but only +1.4% on Counting.
  • InternVL2.5 series: 8B→26B→38B→78B yields 17.4%→20.6%→26.6%→28.6%.
  • Smaller models (MiniCPM-v/o) score highest on text-only queries; adding image context hurts performance.
  • Larger models benefit from video-first ordering; smaller models show negligible difference across orderings.

Limitations

  • Benchmark size is moderate (966 videos, 2,560 QA pairs), which may limit statistical granularity per task.
  • Multiple-choice format with up to 9 options may not capture open-ended generation capabilities.
  • Frame sampling is fixed at 32 frames for all models regardless of video length (≥5 min), potentially disadvantaging models on temporally dense content.
  • The paper notes results are truncated due to space; smaller-model inference pattern ablations are limited to MiniCPM-v/o and InternVL2.5 series.
  • No evaluation of chain-of-thought or tool-augmented inference, leaving open whether prompting strategies close the gap.

Relevance to Vision-Language Models

IV-Bench directly targets a gap in VLM evaluation: the ability to use an external reference image as a visual query anchor for video understanding, analogous to real-world visual search. The benchmark's finding that image context helps large VLMs but hurts small ones suggests that cross-modal grounding is an emergent capability dependent on model scale, with implications for VLM training data curation and architecture design. The result that increasing frame count outperforms resolution scaling provides a concrete practical guideline for video-VLM inference budgeting. The large human–machine gap (28.9% vs. 88.8%) positions this as a useful stress-test alongside existing video-VLM benchmarks like Video-MME and MLVU.