ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ReAgent-V is a multi-agent inference-time harness for video understanding that wraps any base LVLM in a three-stage pipeline: entropy-calibrated frame selection, tool-augmented reasoning by a target agent, and reward-driven multi-perspective reflection by a critic agent. The scalar rewards emitted by the critic during inference are reused as data-quality scores to automatically curate training sets for SFT, DPO, and GRPO, collapsing inference and data-curation into a single pass.

Problem¶

Static single-pass VLMs generate answers without self-correction. Prior fixes either require expensive human annotation, rely on offline pretrained reward models that cannot capture real-time reasoning state, or use static reward templates too rigid for open-ended tasks. Existing multi-model debate and tool-agent frameworks add latency without providing reward signals.

Method¶

Stage 1 — Entropy-Calibrated Frame Selection (ECRS). Each frame \(f_i\) is scored by

\[\text{ECRS}_i = \frac{s_i \cdot H_i}{\sum_{k=1}^{N} H_k}\]

where \(s_i = \frac{e_i^\top q}{\|e_i\|\|q\|}\) is CLIP cosine similarity to the query and \(H_i\) is the mean per-channel RGB entropy of frame \(i\). Frames are iteratively selected using an exponentially tightening threshold \(\text{ECRS}_i > k \cdot \alpha^m \cdot \tau\), suppressing semantically redundant high-CLIP-score frames.

Stage 2 — Tool-Augmented Reasoning. The target agent dynamically selects a subset \(T' \subseteq T\) from a tool factory (OCR, detection, captioning, etc.) given the selected frames and query, producing intermediate outputs \(R_i = T_i(Q, F)\) that inform an initial answer \(A_0\).

Stage 3 — Critic Evaluation and Multi-Perspective Reflection. A critic agent evaluates \(A_0\), generates diagnostic sub-questions, invokes additional tools, and emits a structured evaluation report \(E\) containing a scalar reward and dimension-level feedback (e.g., Visual Alignment, Temporal Accuracy). The target agent then produces three independent revised answers from: - Conservative \(t_c\): adjust only the final answer label. - Neutral \(t_n\): update scene entities based on context. - Aggressive \(t_a\): revise both reasoning steps and entities.

Each produces a confidence score \(p^{(t)} = P(A^{(t)} \mid F, Q, R_\text{update}, E)\). If all three exceed 0.6, the system merges consistent components; otherwise it picks \(\arg\max_t p^{(t)}\). High-scoring inference traces (score \(< 5/10\), indicating harder cases requiring more reflection) are stored as training data for downstream GRPO/DPO.

Key Contributions¶

ECRS metric: jointly accounts for query relevance and frame information density, outperforming CLIP-only selection.
Real-time structured reward: critic generates per-dimension scalar rewards at inference time rather than relying on frozen offline reward models.
Three-strategy reflection: conservative/neutral/aggressive perspectives reduce overconfidence bias and diversify answer candidates.
Inference-as-data-curation: the same evaluation report that guides answer refinement doubles as an automatic difficulty filter for GRPO/DPO training set construction.
Modular harness: target and critic model backends, tool sets, and prompt templates are independently configurable.

Results¶

Video understanding (6 benchmarks, including VideoMME, MLVU, EgoSchema): LLaVA-Video-72B + ReAgent-V gains +6.9% average over the base model; Qwen2.5-VL-72B + ReAgent-V gains +3.2%. LLaVA-Video-72B + ReAgent-V matches GPT-4o (e.g., VideoMME Overall: 73.5 vs. 71.9) using fewer frames.
Frame selection: reduces per-sample inference time by ~20–30% while improving accuracy; e.g., LLaVA-7B on LongBench: 53.1% (21.4 s) vs. 52.6% (36.0 s) without selection.
Video LLM reasoning (GRPO fine-tuning on Qwen2.5-VL-7B): GRPO + ReAgent-V data selection achieves +2.1% over vanilla GRPO using only 52k vs. 116k samples (45% of data); outperforms full 260k SFT by +4.3%.
VLA alignment (OpenVLA-7B on SIMPLER): +9.8% overall over the best baseline (GRAPE template reward); +9.0% on in-domain tasks.
Reflection ablation: enabling the three-perspective mechanism adds 0.2–0.8 pp across benchmarks; aggressive strategy alone has a high correction rate but lower correction accuracy compared to the merged strategy.

Limitations¶

Two-agent inference per sample (target + critic) adds latency not fully recovered by frame selection for all model scales.
Critic agent quality is a ceiling: the evaluation report's fidelity depends on the critic LVLM's own video understanding ability.
ECRS threshold hyperparameters (\(k\), \(\alpha\), \(\tau\)) are fixed; their transferability across video domains is unvalidated.
The data-selection criterion (reward score \(< 5/10\) as "hard") is heuristic and not ablated against alternative cutoffs.
Evaluation confined to existing benchmarks; real-time deployment latency in robotics (VLA) scenarios is not reported.

Relevance to Harnesses / Meta-Harnesses¶

ReAgent-V is a canonical meta-harness: it orchestrates heterogeneous components (frame selector, tool factory, target agent, critic agent) into a deterministic pipeline (Algorithm 1) that wraps arbitrary base LVLMs without modifying their weights at inference time. The dual-use of inference traces—simultaneously refining answers and curating training data—exemplifies the harness pattern of treating runtime execution as a data flywheel, a design increasingly common in self-improving agent systems. Its modular separation of tool invocation, reward generation, and reflection strategies makes it directly comparable to tool-agent orchestration frameworks like LangGraph or AutoGen, but specialized for video with a structured reward loop. For researchers tracking harness design, ReAgent-V illustrates how a structured critic-in-the-loop reflection harness can replace both human annotation pipelines and offline reward models.