Skip to content

ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReAgent-V is a multi-agent video understanding framework that generates real-time reward signals during inference to guide iterative answer refinement and automatic high-quality data selection. It pairs a target agent with a critic agent that scores answers and drives multi-perspective reflection (conservative/neutral/aggressive), enabling both better inference-time performance and downstream fine-tuning without costly human annotation.

Problem

Existing LVLMs for video understanding use static, single-pass reasoning with no self-correction loop. Offline reward models fail to capture real-time reasoning states; reward templates are predefined and cannot handle open-ended evaluation; annotation for high-quality video training data is prohibitively expensive. Prior multi-agent/debate approaches improve reasoning but lack inference-time reward signals and efficient frame selection.

Method

ReAgent-V operates in three stages:

  1. Entropy-Calibrated Frame Selection (ECRS): Scores each frame by the product of its CLIP-query cosine similarity and per-channel RGB entropy, normalized across the candidate set. Frames are iteratively selected using an exponentially increasing threshold, suppressing redundant high-CLIP-score but semantically irrelevant frames.

  2. Tool-Augmented Reasoning: The target agent selects a task-appropriate tool subset (OCR, object detection, captioning, ASR, etc.) from a modular tool factory, applies them to selected frames and the query, and produces an initial answer Aâ‚€.

  3. Evaluation and Multi-Perspective Reflection: A critic agent evaluates Aâ‚€; if unsatisfactory, it generates diagnostic sub-questions, invokes additional tools, and produces a structured evaluation report with scalar sub-scores (visual alignment, temporal accuracy, etc.). The target agent then generates three reflected answers from conservative (final answer only), neutral (update scene entities), and aggressive (revise reasoning steps + entities) perspectives, each with a confidence score. If all confidence scores exceed 0.6, the final answer merges common components across perspectives; otherwise the highest-confidence revision wins.

The evaluation reports also serve as importance scores for automatic data curation: low-scoring (harder) samples are retained for GRPO fine-tuning, bypassing manual annotation.

Key Contributions

  • ECRS frame selector combining semantic relevance and information entropy for efficient, non-redundant keyframe selection.
  • Real-time process reward generated by a critic agent during inference, replacing offline reward models or static templates.
  • Multi-perspective reflection mechanism (conservative/neutral/aggressive) to mitigate overconfidence bias and improve answer robustness.
  • Automatic data pipeline: inference-time reward scores filter high-quality samples for SFT, DPO, and GRPO without human labeling.
  • VLA alignment application: ReAgent-V replaces template-based reward functions for trajectory-wise preference optimization of robot action models.
  • Modular, lightweight architecture supporting flexible base-model and tool customization.

Results

  • Video understanding (6 benchmarks: LongBench, NextQA, EgoSchema, LVBench, MLVU, VideoMME):
  • Qwen2.5-VL-72B + ReAgent-V: +3.2% avg over base; LLaVA-Video-72B + ReAgent-V: +6.9% avg over base.
  • LLaVA-Video-72B + ReAgent-V reaches GPT-4o-comparable performance with fewer input frames.
  • Outperforms VideoMemAgent and VideoAgent on all reported benchmarks with lower inference time.
  • Video LLM reasoning (6 benchmarks; Qwen2.5-VL-7B base):
  • GRPO + ReAgent-V data selection: +2.1% over vanilla GRPO using only 52k vs. 116k samples (45% of data).
  • +4.3% over full 260k-sample SFT.
  • VLA alignment (SIMPLER environment, OpenVLA-7B):
  • +9.8% overall over best baseline (GRAPE template reward); +9.0% on in-domain task.
  • Frame selection ablation (Table 3): selection reduces per-sample inference time (e.g., LLaVA-72B: 83→68 s on LongBench) while improving accuracy.
  • Reflection ablation (Table 4): consistent +0.2–0.8% across models on all three benchmarks; aggressive-only strategy underperforms due to over-modification of correct reasoning.

Limitations

  • Inference involves a critic agent pass plus three reflection passes, adding latency despite frame selection savings—not analyzed end-to-end against single-pass models at equal wall-clock budget.
  • Confidence threshold (0.6) and ECRS hyperparameters (Ï„, α, k) require tuning; sensitivity not reported.
  • Reflection benefit is modest at 72B scale (≤0.5% gains in Table 4), suggesting diminishing returns for larger base models.
  • VLA evaluation is limited to simulation (SIMPLER); real-world robot transfer not demonstrated.
  • Data selection strategy retains only low-scoring (score < 5/10) samples; the choice of this threshold and its interaction with domain shift are not analyzed.

Relevance to Vision-Language Models

ReAgent-V directly addresses a core bottleneck in video LVLMs: the inability to self-correct during inference without expensive human-annotated reward signals. The critic-agent design provides a generalizable recipe for process-level reward generation that can plug into any LVLM backbone (demonstrated across LLaVA-Video and Qwen2.5-VL at 7B–72B). For VLM researchers, the dual use of inference-time signals—both for answer refinement and for automatic training data curation—is a concrete technique for reducing annotation cost in video domains. The framework also extends the VLM-to-VLA bridge, showing that LVLM-based reward models can outperform hand-crafted reward templates in robot alignment, a growing concern in embodied AI.