Skip to content

VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understanding

🕒 Published (v1): 2025-12-13 15:11 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoARM is a training-free agentic framework for long-form video understanding that replaces static, exhaustive preprocessing with adaptive coarse-to-fine reasoning over a three-tier hierarchical multimodal memory (HM³). An MLLM controller iterates through observe–think–act–memorize loops, invoking complementary temporal scoping and multimodal understanding tools on-demand. It outperforms prior state-of-the-art (DVD) across five benchmarks while consuming only 1/34 the tokens experimentally.

Problem

Existing long-form video understanding methods either rely on hand-crafted, rigid reasoning pipelines (e.g., VideoTree, LLoVi) that limit MLLM autonomy, or pre-build exhaustive clip-level databases via dense MLLM captioning (e.g., DVD) that are token-intensive, query-agnostic, and cannot be updated during inference. Neither approach fully exploits the native visual reasoning and planning capacity of modern MLLMs under strict token budgets.

Method

VideoARM consists of two coupled components:

Hierarchical Multimodal Memory (HM³) — three tiers updated continuously during agent execution: - Sensory Memory: long-term perception pool (current focused frame interval, stored as 3×2 image grids) and short-term perception pool (fine-grained local clips + audio for hypothesis testing, cleared after each cycle). - Result Memory: temporally ordered log of tool outputs (interval, caption/Q&A, confidence) for reflection and strategy adaptation. - Working Memory: controller reasoning traces and intents, externalized from the MLLM context to prevent overflow and enable auditable history.

Coarse-to-Fine Video Reasoning Agent — a controller (OpenAI o3) operating in a ReAct-style loop grounded by HM³, selecting from two toolsets: - Temporal Scoping Tools: Interval Localizer (identifies query-relevant frame intervals, samples N₁ ∈ [30,150] frames adaptively, updates long-term pool) and Clip Explorer (tentative local probing without changing global focus, stores short clips + audio in short-term pool). - Multimodal Understanding Tools: Scene Snapper (coarse global captioning over long-term pool), Audio Transcriber (whisper-1 on short-term audio), Clip Analyzer (fine-grained local Q&A with confidence score over short-term pool).

The controller reads HM³, reasons over missing evidence, invokes one tool per step, writes results back to memory, and iterates up to N=10 steps before emitting a final answer.

Key Contributions

  • A training-free agentic paradigm (VideoARM) that performs query-driven, on-the-fly video reasoning without any offline database construction or model fine-tuning.
  • HM³: a dynamically constructed three-tier hierarchical multimodal memory that externalizes MLLM context, reduces redundancy, and supports coarse-to-fine evidence accumulation.
  • Complementary toolsets (Temporal Scoping + Multimodal Understanding) enabling both broad temporal exploration and precise local verification within the same loop.
  • Demonstrated 1/34× token consumption versus DVD experimentally on Video-MME while achieving higher accuracy.

Results

  • Video-MME (w/o subtitles): VideoARM (o3+GPT-4o): 81.3 / 81.2 / 76.4 / 82.8% (short/medium/long/overall) vs. DVD's reported 71.6% overall; surpasses all listed commercial VLMs (GPT-4o: 65.3% long, Gemini-1.5-Pro: 67.4% long) and open-source VLMs on long subset.
  • LongVideoBench (long subset): VideoARM 78.0% vs. DVD 73.0%; VideoARM (o3+GPT-4.1) 73.7%.
  • EgoSchema: VideoARM (o3+GPT-4o) 85.8%, (o3+GPT-4.1) 86.4% vs. DVD 76.6%.
  • MLVU: VideoARM (o3+GPT-4.1) 81.2% vs. VideoLucy 76.1%, GPT-4o 64.6%.
  • LVBench: VideoARM (o3+GPT-4.1) 79.7% vs. DVD 76.0%, VideoLucy 58.8%.
  • Token efficiency: ≤80K tokens per query vs. DVD's ≥3.98M for a 30-minute video (~1/50 theoretical; 1/34 experimental on Video-MME).
  • Ablation: removing long-term perception pool causes −9.5% on Video-MME (long); removing Result Memory causes invalid/repeated cycles; adaptive frame sampling (avg N₁=49.8) outperforms fixed N₁=60 on both long-video benchmarks.

Limitations

  • Depends on proprietary APIs (OpenAI o3, GPT-4o/4.1, whisper-1); results are not reproducible with open-source models only — ablations show Qwen3-VL and GPT-4o as controller score 54.9% and 40.5% vs. o3's 76.5%–80.0%.
  • Step budget N=10 provides diminishing returns on short videos (marginal gains, possible redundancy).
  • Token efficiency analysis for DVD excludes caption generation and retrieval tokens, making the comparison a conservative lower bound rather than a full end-to-end comparison.
  • No fine-tuning means the system's ceiling is bounded by the base MLLM's capabilities; performance gaps between controller choices are large (~35 percentage points).
  • Evaluation is limited to multiple-choice QA benchmarks; open-ended generation or temporal grounding tasks are not assessed.

Relevance to Agentic AI / LLM Agents

VideoARM is a strong instantiation of the ReAct-style agentic loop applied to a challenging perception-heavy domain, demonstrating that externalizing memory into a structured, hierarchically organized store (rather than relying on in-context accumulation) substantially improves both performance and token efficiency. The three-tier HM³ design—separating raw perception, intermediate results, and reasoning traces—is a transferable architectural pattern for any long-horizon agentic task where context overflow is a bottleneck. The paper directly validates the trade-off between static preprocessing (database-centric agents like DVD) and adaptive on-the-fly reasoning, providing empirical evidence that query-guided tool invocation outperforms exhaustive precomputation. For researchers tracking agentic AI, VideoARM advances the conversation on memory management, tool design, and controller autonomy in multimodal settings.