RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; RL with verifiable meta-reasoning rewards for robust long-horizon agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RLVMR augments end-to-end RL for LLM agents with dense, verifiable rewards for explicit meta-reasoning behaviors (planning, exploration, reflection, monitoring), directly supervising the reasoning process rather than only the final outcome. On ALFWorld and ScienceWorld, a 7B RLVMR agent achieves 83.6% on the hardest out-of-distribution split (L2), surpassing much larger proprietary models while substantially reducing redundant and invalid actions.

Problem

Outcome-only RL (e.g., GRPO) for long-horizon agents suffers from inefficient exploration: agents are rewarded for any successful trajectory, including those built on redundant, repetitive, or logically flawed action sequences. This produces policies that generalize better than SFT but with high invalid-action rates (up to 31.2% repetitive actions at L2 with 7B GRPO). SFT is efficient but brittle; GRPO generalizes but reinforces unsound reasoning. Neither paradigm supervises how the agent reasons, only whether it succeeds.

Method

RLVMR is a two-phase framework:

Phase 1 — Cold-start SFT. A teacher model (GPT-4) annotates 200 successful trajectories with four XML-style meta-reasoning tags: <planning>, <explore>, <reflection>, and <monitor>. The target LLM is fine-tuned on these 200 annotated examples to learn tag syntax.

Phase 2 — GRPO-MR (online RL). The agent generates structured outputs <tag>...</tag><action>...</action> at each step. A composite reward is computed: - Outcome reward \(R(\tau)\): binary success signal at trajectory end. - Meta-reasoning reward \(r_t^{MR}\): step-level dense signals — <planning> rewarded if the trajectory succeeds; <explore> rewarded if the action targets a novel object/location; <reflection> rewarded if it immediately precedes a corrective action after failures. - Format penalty \(-\lambda_{\text{format}}\) for malformed output structure.

The per-step advantage combines trajectory-level normalization and within-tag-group normalization: $\(A_t = \alpha \cdot A_k^{\text{traj}} + (1-\alpha) \cdot A_{t,\text{tag}}^{MR}\)$ where \(A_k^{\text{traj}} = (R(\tau_k) - \mu_R)/\sigma_R\) and \(A_{t,\text{tag}}^{MR} = (r_{t,\text{tag}}^{MR} - \mu_{\text{tag}})/\sigma_{\text{tag}}\).

Policy is optimized with a clipped surrogate + KL penalty against a reference policy: $\(\mathcal{L} = \mathbb{E}_t[\min(r_t(\theta)A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t)] - \lambda_{KL} D_{KL}(\pi_\theta \| \pi_{\text{ref}})\)$

Key Contributions

• Identifies and quantifies inefficient exploration in outcome-only RL: scaling model size from 1.5B to 7B worsens repetitive action rate under GRPO (27.1% → 31.2% on L2), showing the deficiency is in the training objective, not model capacity.
• Proposes RLVMR: a framework that operationalizes metacognitive theory into verifiable, programmatic process rewards without learned critics.
• Introduces GRPO-MR: a critic-free policy gradient method with dual-level advantage (trajectory + within-tag-group normalization) to jointly optimize task success and reasoning quality.
• Achieves SOTA on ALFWorld and ScienceWorld across seen and unseen task splits with only 200 cold-start SFT examples.

Results

ALFWorld (success rate %): - RLVMR 7B: L0 91.4 / L1 91.8 / L2 83.6 vs. GiGPO 7B (89.5 / 90.2 / 67.2), GRPO 7B (79.3 / 77.3 / 52.3), GPT-4o ReAct (57.3 / 66.0 / 68.8), DeepSeek-R1 ReAct (68.8 / 70.2 / 67.3) - RLVMR 1.5B: L2 56.3% vs. GRPO 1.5B 29.7% - RLVMR Llama3.1-8B: L2 83.2% vs. GiGPO 68.8%

ScienceWorld (success rate %): - RLVMR 7B: L0 67.2 / L1 43.0 / L2 32.2 vs. GiGPO 7B (53.4 / 35.2 / 25.8), GRPO 7B (49.1 / 30.1 / 26.6)

Reasoning efficiency (ALFWorld, 7B): - RLVMR reduces repetitive action rate to 2.3% (L0), 11.7% (L2) vs. GRPO's 20.3% (L0), 31.2% (L2) - Invalid action rates also drop across all splits vs. SFT and GRPO

Limitations

• Cold-start SFT requires a powerful teacher model (GPT-4) for trajectory annotation, introducing a dependency on closed proprietary models.
• Meta-reasoning reward design is task-type-specific (e.g., novelty detection for exploration); generalization to domains beyond embodied/text-based household and science tasks is unverified.
• The monitoring tag (<monitor>) has no explicit reward signal defined in the presented framework — its contribution is unclear.
• Evaluation confined to two text-based benchmarks (ALFWorld, ScienceWorld); real-world or visual environments are not assessed.
• The \(\alpha\) balancing hyperparameter and reward scaling require tuning; sensitivity analysis is not fully presented in the provided text.

Relevance to Agentic AI / LLM Agents

RLVMR directly addresses one of the central open problems in LLM agent training: that sparse outcome rewards are insufficient to instill coherent reasoning, producing agents that succeed by coincidence rather than by design. By treating metacognitive skills (planning, exploration, reflection) as first-class, verifiable reward signals — rather than opaque internal states — RLVMR offers a practically deployable alternative to process reward models that require learned critics. The result is directly relevant to any long-horizon agent training pipeline, demonstrating that process-level supervision substantially improves out-of-distribution generalization even with a tiny SFT corpus (200 trajectories). This work also contributes an important diagnostic — the inefficient exploration framing with concrete metrics — that may become a standard evaluation lens for RL-trained agents alongside task success rate.