Skip to content

ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReAgent-V is a multi-agent video understanding framework that generates real-time reward signals during inference via a critic agent, which drives multi-perspective answer refinement (conservative/neutral/aggressive) and automatically filters high-quality training data for SFT, DPO, and GRPO. Tested across 12 datasets, it improves video understanding by up to 6.9%, video LLM reasoning by 2.1%, and VLA alignment by 9.8% over strong baselines.

Problem

Existing video LVLMs use a single-pass, static reasoning paradigm with no self-correction. Prior attempts at reward-based RL rely on offline reward models that cannot capture the model's real-time reasoning state, while tool-agent frameworks lack reward signals and are inference-inefficient. High-quality annotation for SFT remains expensive and unscalable.

Method

ReAgent-V operates in three stages at inference: 1. Entropy-Calibrated Frame Selection: An iterative algorithm scores frames with ECRS = (cosine CLIP similarity to query) × (per-frame RGB entropy) / (total entropy), selecting frames that are both semantically relevant and informationally rich, suppressing redundant frames that CLIP-only selection admits. 2. Tool-Augmented Reasoning: A target agent dynamically selects a subset of tools (OCR, ASR, captioning, detection) and produces an initial answer A₀ from selected frames + tool outputs. 3. Critic Evaluation and Multi-Perspective Reflection: A critic agent evaluates A₀, generates clarifying sub-questions, invokes additional tools, and produces a scalar reward score plus structured feedback report E. The target agent then revises from three stances—conservative (change only final answer), neutral (update scene entities), aggressive (revise reasoning steps and entities)—each yielding a revised answer and confidence score. If all confidences exceed 0.6, answers are merged; otherwise the highest-confidence answer is selected. High-scoring inference traces are stored to automatically curate SFT/DPO/GRPO training data.

Key Contributions

  • Entropy-Calibrated Relevance Score (ECRS) for efficient, question-relevant keyframe selection without reliance on LLM-based captioning.
  • Inference-time critic agent that produces process rewards (multi-dimensional scalar scores + structured feedback) without pre-trained offline reward models.
  • Three-perspective reflection mechanism (conservative/neutral/aggressive) that reduces overconfidence errors while enabling targeted answer correction.
  • Unified pipeline that simultaneously improves inference performance and automatically mines high-quality training data, demonstrated across video understanding, video LLM reasoning (GRPO with 45% of data), and VLA alignment (TPO with LLM-as-reward-model).

Results

  • Video Understanding: ReAgent-V + Qwen2.5-VL-72B achieves 75.1% on VideoMME (vs. 74.1% Qwen2.5-VL-72B baseline; comparable to GPT-4o at 71.9% overall); LLaVA-Video-72B + ReAgent-V improves from 67.1% → 73.5% on VideoMME overall (+6.9% average gain across benchmarks).
  • Video LLM Reasoning: GRPO + ReAgent-V (52k samples) outperforms vanilla GRPO (116k) by 2.1% relative average and SFT on 260k by 4.3%, across VSI-Bench, VideoMMMU, MMVU, MVBench, TempCompass, VideoMME.
  • VLA Alignment: ReAgent-V as reward model for TPO on OpenVLA-7B in SIMPLER outperforms GRAPE (template reward) by 9.8% overall and 9.0% on original tasks.
  • Frame Selection: Reduces per-sample inference time by ~20–25% vs. no selection while improving accuracy across LLaVA-7B, QwenVL-7B, LLaVA-72B, Qwen2.5-72B on LongBench, VideoMME, EgoSchema.
  • Reflection: Adding reflection module yields consistent gains of 0.1–0.9% across models; conservative strategy has higher correction accuracy than aggressive.

Limitations

  • Inference overhead from the critic agent and multi-perspective reflection adds latency, even if frame selection partially offsets this.
  • The critic and target agent are set to the same model in experiments; performance with weaker critics is not characterized.
  • Aggressive reflection strategy lowers correction accuracy by erroneously modifying previously correct reasoning steps, which the merging mechanism only partially mitigates.
  • VLA experiments are conducted in simulation (SIMPLER) only; real-world robot transfer is not evaluated.
  • Data selection for GRPO relies on keeping low-scoring samples (score < 5/10), an empirically motivated threshold that may require tuning per domain.

Relevance to Agentic AI / LLM Agents

ReAgent-V is a concrete instantiation of inference-time agentic self-correction: a critic agent generates process rewards on-the-fly—without a separate trained reward model—closing the loop between evaluation and refinement within a single inference pass. The multi-perspective reflection mechanism (conservative/neutral/aggressive) is a novel strategy for reducing agent overconfidence, directly relevant to the challenge of reliable LLM agent behavior. Its dual use as both an inference-time reasoner and an automatic data-quality filter for RL training (GRPO, DPO) makes it a practical example of using agent-generated signals to bootstrap stronger agents. The modular, tool-augmented design with dynamic tool selection is a worked example of how agentic video systems can be composed from LVLM backbones without retraining.