Skip to content

MR. Video: MapReduce as an Effective Principle for Long Video Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MR. Video applies the MapReduce principle from distributed systems to long video understanding: parallel Map steps densely caption short segments while Reduce steps aggregate segment-level analyses into a global answer. Two novel designs—consistent character naming across captions and question intention analysis—address gaps in prior video agents. On LVBench, it surpasses all prior VLMs and video agents by >7% accuracy.

Problem

Sequence-to-sequence VLMs are bottlenecked by context length, forcing sparse frame sampling that misses local detail. Video agents bypass context limits but rely on sequential key-frame retrieval, which (a) breaks sequence-parallel scalability and (b) fails when queries require global context (e.g., counting 200 events, identifying a protagonist across an hour-long video). Neither paradigm simultaneously achieves dense local perception, unlimited context, and inference scalability.

Method

MR. Video instantiates three MapReduce stages using a text LLM (GPT-4o) and a vision LLM (Gemini-2.0-Flash):

  1. Captioning (MapReduce stage 1): Map independently captions short scenes (checking clip continuity, extracting key characters with frame indices). Reduce merges character/object identities across segments via VLM-based association and renames them with a consistent <entity>_<index> schema, producing 500–2k captions per hour-long video.

  2. Question Intention Analysis (MapReduce stage 2): Map processes each segment's captions + midframe images to infer which scenes may answer the question, outputting reasoning, candidate scenes, and unresolved key subjects. Reduce aggregates all segment analyses into a video-level localization, explicitly recovering implicit question context (who is the protagonist, what room is the "utility room") rather than performing embedding-based retrieval.

  3. Goal-Aware Analysis (MapReduce stage 3): Map issues LLM-proposed VLM queries against candidate scenes, sampling frames densely (local visual detail) or sparsely across segments (temporal reasoning). Reduce merges potentially contradictory scene-level answers into a final response.

Context length per VLM call is capped at 40 frames. Computational complexity is reduced from O(N²) (monolithic VLM) to O(N²/M) where M is the number of parallel segments.

Key Contributions

  • Formalises MapReduce (parallel Map + aggregating Reduce) as a principled framework for long video understanding, addressing context-length limits, sequential bottlenecks, and global context deficits simultaneously.
  • Consistent character/object naming via VLM-based cross-segment entity merging, enabling coherent multi-hop reasoning over long temporal spans.
  • Question intention analysis that reasons jointly over full-video captions and questions to localise relevant scenes, replacing brittle embedding-based key-frame retrieval.
  • State-of-the-art accuracy on LVBench (+7% over next-best) and consistent improvement over the base VLM (Gemini-2.0-Flash) on LongVideoBench, EgoSchema, and Video-MME, while using smaller per-call context than compared VLMs.

Results

  • LVBench overall: MR. Video 60.8% vs. AdaReTaKe 53.3% (previous SOTA VLM), VCA 41.3% (best prior video agent), Gemini-2.0-Flash 48.6%, GPT-4o 48.9%.
  • LVBench sub-tasks: KIR (key information retrieval) 71.4%, ER (entity recognition) 59.8%, TG (temporal grounding) 58.8%.
  • LongVideoBench (Val Long): MR. Video 61.6% vs. VCA 73.6% (N/A — VCA not evaluated on this split in paper), VideoAgent 47.6%, Gemini-2.0-Flash 45.7%.
  • EgoSchema: MR. Video 73.8% vs. VideoTree 67.0%, GPT-4o 70.4%.
  • Video-MME (Long, no subtitles): MR. Video 63.4% vs. VideoAgent 46.4%, GPT-4o 65.3%.
  • Ablation (LVBench 6-video subset): removing consistent character names drops from 63.3% to 34.4%; removing goal-aware analysis drops to 52.0%.
  • Key-frame recall: question intention analysis localises target scenes for 70.4% of questions vs. 34.4% for MM-Embed retrieval.
  • Cost: ~\(0.80 Gemini-2.0-Flash per hour-long video for captioning; ~\)0.40 GPT-4o per LVBench question.

Limitations

  • Relies on proprietary API models (GPT-4o, Gemini-2.0-Flash), making cost and reproducibility dependent on third-party services (~$0.40/question).
  • Counting accuracy is still imperfect: MR. Video under-counts events when strict checking criteria miss valid instances (e.g., counts 78 Salah goals vs. ground truth 200).
  • Ablation uses only 6 videos/98 questions due to budget constraints, potentially under-powering component analysis.
  • No training or fine-tuning; prompt engineering may be brittle across domains not represented in LVBench.
  • System-level parallelism benefits (O(N²/M) scaling) are demonstrated analytically but not empirically benchmarked in wall-clock experiments.

Relevance to Vision-Language Models

MR. Video directly targets a core limitation of VLMs—context-length constraints in long-video settings—by decoupling dense visual perception (VLM) from global reasoning (LLM) via a principled MapReduce decomposition rather than end-to-end sequence processing. The consistent character naming technique addresses a known failure mode in current VLMs: inability to maintain entity identity across long temporal spans without explicit cross-segment state. The question intention analysis paradigm challenges the prevalent paradigm of embedding-based frame retrieval used in many VLM-based agents, demonstrating that full-context language reasoning outperforms local similarity matching (70.4% vs. 34.4% recall). For VLM researchers, this work establishes that agentic orchestration of off-the-shelf VLMs can exceed dedicated long-video VLMs, pointing toward system-level design as a productive research axis.