Skip to content

LongVideoAgent: Multi-Agent Reasoning with Long Videos

🕒 Published (v1): 2025-12-23 18:59 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LongVideoAgent is a multi-agent framework for long-form video QA in which a master LLM coordinates a grounding agent (temporal localization) and a vision agent (frame-level perception), trained with GRPO reinforcement learning to emit concise, structured action tokens. It introduces two episode-level benchmarks (LongTVQA/LongTVQA+) built by aggregating TVQA clips to hour-scale sequences. The system significantly outperforms non-agentic baselines and narrows the open-source/closed-source gap.

Problem

Existing multimodal LLMs process long videos via lossy downsampling or coarse summarization, collapsing temporal reasoning into an irreversible early step that cannot recover fine-grained visual evidence. Prior agentic approaches (VideoAgent-style) relied on weak toolsets and lacked multi-step planning or RL-driven coordination, leaving a gap between efficiency, temporal grounding precision, and multimodal completeness.

Method

A master agent (LLM) runs a bounded loop of at most K steps, emitting exactly one structured action token per turn: <visual_query> (calls the vision agent), <request_grounding> (calls the grounding agent), or <answer> (terminates). The grounding agent receives the question and full episode subtitles and returns a symbolic clip tag <clip_X>. The vision agent receives the clip tag plus an on-demand prompt and returns textual observations (objects, OCR, scene cues) from frames within that segment — no raw images reach the master. Accumulated context (subtitles + clip tags + visual observations) grows iteratively until the master terminates. For open-source masters, the policy is fine-tuned with GRPO using two rule-based rewards: (i) structural validity (exactly one well-formed action tag per turn) and (ii) exact-match answer correctness at termination. Vision and grounding agents remain frozen during RL.

Key Contributions

  • Three-agent architecture (master + grounding + vision) with a structured action-token protocol separating temporal localization from fine-grained visual extraction.
  • GRPO-based RL training for the master agent using only format + answer-correctness rewards, producing interpretable multi-turn reasoning traces.
  • LongTVQA and LongTVQA+ benchmarks: episode-level (hour-scale) aggregations of TVQA/TVQA+ with re-indexed timestamps and preserved bounding-box annotations.
  • Demonstration that open-source AgenticRL-Qwen2.5-7B matches closed-source GPT-5-mini on LongTVQA.

Results

  • AgenticRL-Qwen2.5-7B: 60.20% LongTVQA / 70.80% LongTVQA+ vs. non-agentic 46.10% / 60.30% (+14.10 / +10.50).
  • AgenticRL-Qwen2.5-3B: 47.40% / 50.10% vs. non-agentic 23.50% / 27.70% (+23.90 / +22.40) — largest relative gain.
  • Agentic-Grok: 82.65% / 85.60% vs. non-agentic 76.90% / 81.80% (+5.75 / +3.80), best overall.
  • Grounding alone adds +4.7 accuracy points over text-only; adding the vision agent adds another +5.8 (Table 4a).
  • Increasing window context from 1 to 3 clips raises accuracy from 70.33% to 77.26% and grounding from 71.67% to 81.94%.
  • GPT-4o as vision agent outperforms Qwen3-VL-235B by +4.33 accuracy points.
  • Step budget K=5 is sufficient; K=10 yields no additional accuracy gain.

Limitations

  • Relies on provided subtitle tracks; raw audio/ASR is not processed.
  • Grounding and vision agents are kept frozen during RL; joint optimization is unexplored.
  • Reward function is minimal (format + exact match), leaving room for denser reward shaping.
  • Evaluation limited to TVQA-derived TV-show content; generalization to other long-video domains is untested.
  • Larger window contexts increase latency without proportional accuracy gains.

Relevance to Harnesses / Meta-Harnesses

LongVideoAgent is a domain-specific meta-harness: a master controller that dynamically dispatches specialized sub-agents (grounding, vision) under a step budget, accumulates their outputs into a shared context, and terminates based on sufficiency judgment — the same orchestrator-plus-specialist pattern central to general meta-harness design. The GRPO training scheme for the master while keeping sub-agents frozen is a direct analog to training an orchestrator policy on top of fixed tool APIs, which is a key open problem in harness RL. The structured action-token protocol (exactly one tag per turn, strict format rewards) provides a concrete example of enforcing harness-level control flow via reward shaping rather than hard-coded logic.