ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents¶

🕒 Published (v1): 2026-06-30 09:19 UTC · Source: Arxiv · link

Why this paper was selected

ReGRPO for tool-use VLMs; reflection+policy optimization is the timely RL-for-agents paradigm

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ReGRPO introduces a reflection-augmented extension of GRPO for tool-using VLM agents, where structured Reflection-of-Thought (RoT) triplets are both supervised (warm-start SFT) and RL-optimized jointly with corrective actions. A deterministic zero-verifier trigger at inference opens a local reflection-correction block only on detected failure, requiring no external verifier calls. On GTA and GAIA, ReGRPO achieves the best results among compared open-source controllers under identical backbone and tool configurations.

Problem¶

Two persistent gaps afflict tool-using VLM agent training: (1) SFT on successful trajectories provides no recovery signal when the agent deviates mid-trajectory (e.g., layout shift causes an empty OCR return), so errors cascade without grounded guidance; (2) standard RL with sparse, trajectory-level rewards cannot localize which step failed or prescribe a specific repair, making learning slow and unstable in long-horizon multi-step tasks. Methods such as MAT-AGENT (SFT-only) and SPORT (step-wise preference RL) each address one gap partially but lack structured, causally grounded failure diagnostics that can be optimized end-to-end.

Method¶

ReGRPO combines three components:

1. Structured Reflective Data Engine. Starting from ground-truth step $a^*_i$ (from MM-Traj), the engine synthesizes "near-miss" actions $a^{fail}_i$ by perturbing tool choices or arguments (e.g., shifted bounding boxes, wrong table columns). These are executed in a sandbox to obtain real failure observations $o^{fail}_i$. A teacher VLM (GPT-4o, offline only) then generates a constrained RoT triplet $z_i = (\text{ErrorType}, \text{Evidence}, \text{FixPlan})$ grounded in $o^{fail}_i$, paired with the corrected action $a^*_i$ to form reflective supervision tuples $\tau^{reflective}_i = (x_i, a^{fail}_i, o^{fail}_i, z_i, a^*_i)$. Warm-start SFT maximizes $\log P_\theta(z_i, a^*_i \mid x_i, a^{fail}_i, o^{fail}_i)$.

2. Reflection-Augmented GRPO Training. Local trajectory candidates take the form $\tau_i = (a^{(0)}_i, o^{(0)}_i)$ on success, or $(a^{(0)}_i, o^{(0)}_i, z_i, a^{(1)}_i, o^{(1)}_i)$ when reflection is triggered. A composite reward is: $$R(\tau) = \lambda_{exec} \mathbf{1}_{\text{success}} - \eta\, C(\tau) + \lambda_{val}\, V(x_i, \tau)$$ where $C(\tau)$ penalizes reflection token length (forcing parsimonious reflection), and $V(\cdot)$ is an optional training-only verifier score computed deterministically from RoT metadata (no in-loop LLM calls; disabled at inference). Group-relative advantage $A^{(k)}_i = R(\tau^{(k)}_i) - \bar{R}_i$ directly scales gradients on reflection tokens since $\log \pi_\theta(z_i \mid \cdot)$ is factorized within the trajectory likelihood. The optimizer is standard GRPO with a KL penalty to a reference policy.

3. Zero-Verifier Inference. A deterministic gate on normalized tool outputs opens a reflection-correction block only when a hard failure is detected (e.g., empty OCR, API exception); otherwise inference continues single-path. No verifier or external LLM is queried at deployment.

Key Contributions¶

Structured Reflective Data Engine: converts tool-execution failures into grounded RoT supervision triplets (ErrorType, Evidence, FixPlan) paired with corrected actions via realistic near-miss failure induction.
ReGRPO: extends GRPO with structured reflection trajectory parameterization and a reflection-cost reward term, jointly optimizing reflection and corrective action tokens under group-relative advantages.
Zero-Verifier Inference Protocol: deterministic single-path trigger for local reflection-correction blocks, requiring no in-loop LLM calls and enabling efficient deployment.
Strongest open-source controller results on both GTA and GAIA benchmarks under identical backbone and tool suite.

Results¶

(Paper text truncated before the quantitative results table; the following is taken from claims stated in the text.)

ReGRPO "consistently outperforms strong open-source baselines" (MAT-AGENT, SPORT) on GTA and GAIA under the same backbone and tool suite.
Achieves "the best results among the compared open-source controllers" on both benchmarks.
Specific accuracy numbers are not present in the provided text.

Limitations¶

Warm-start SFT and offline RoT annotation depend on GPT-4o, introducing a teacher cost and potential quality ceiling; errors in teacher-generated reflections propagate into training data.
The deterministic failure trigger relies on normalized tool-output heuristics that may not generalize to all failure modes, particularly silent semantic errors that do not produce hard exception signals.
Reflection is constrained to a single local correction block per step; cascading or multi-hop failures may require deeper recursive repair not modeled here.
Near-miss failure synthesis is derived from MM-Traj perturbations, so the failure distribution may not cover out-of-distribution tool errors encountered in deployment.
Quantitative results are not fully provided in the text excerpt, making it impossible to assess margin of improvement or statistical significance.

Relevance to Agentic AI / LLM Agents¶

ReGRPO directly targets a core open problem in agentic systems: how to teach an agent to recover from its own intermediate failures rather than requiring an expert rollout or a scalar outcome signal. By treating reflection as a learnable policy variable rather than a fixed inference-time prompt, it connects to the broader move from frozen chain-of-thought heuristics toward end-to-end-trained metacognition. The zero-verifier inference design is practically significant for deployment, since most real agentic pipelines cannot afford per-step verifier calls. The structured RoT triplet formalism (ErrorType, Evidence, FixPlan) offers a reusable diagnostic abstraction that could be applied beyond VLMs to any tool-calling agent trained with group-relative policy optimization.