Agentic Reinforced Policy Optimization¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RLVR extended to agentic multi-step tasks; bridges single-turn RL to agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ARPO (Agentic Reinforced Policy Optimization) is a novel RL algorithm for training multi-turn LLM tool-use agents that adaptively branches rollout sampling at high-entropy tool-call steps instead of relying solely on trajectory-level sampling. It couples this with advantage attribution estimation to assign fine-grained credit to shared vs. branched token segments. Across 13 benchmarks, ARPO outperforms trajectory-level baselines (GRPO, DAPO, REINFORCE++) by ~4% average accuracy while requiring only half the tool-call training budget.

Problem¶

Standard RLVR algorithms (GRPO, DAPO, REINFORCE++) use trajectory-level rollout sampling: they sample complete tool-use trajectories and assign rewards only from final outputs, ignoring uncertainty at intermediate tool-call steps. This neglects a key empirical observation — token entropy spikes sharply in the first 10–50 tokens after each tool-call result — leaving latent step-level tool-use behaviors underexplored and causing poor sample efficiency in multi-turn agentic settings.

Method¶

ARPO extends the standard rollout with two components:

Entropy-based Adaptive Rollout: Given a global rollout budget $M$, the policy first generates $N$ trajectory-level samples. After each tool-call step $t$, it computes a $k$-token entropy matrix $H_t$ and measures normalized entropy variation $\Delta H_t = \text{Normalize}(H_t - H_{\text{initial}})$. A branching probability is then computed as: $$P_t = \alpha + \beta \cdot \Delta H_t$$ If $P_t > \tau$, the model spawns $Z$ partial rollout branches from that node to explore alternative tool-integrated reasoning paths, consuming from the remaining budget $M - N$.

Advantage Attribution Estimation (hard vs. soft): Tokens on a shared prefix across branched trajectories receive a pooled advantage $\hat{A}^{\text{shared}}_{i,t} = \frac{1}{d}\sum_{i=1}^d \hat{A}_{i,t}$, while tokens on diverged branches receive individual normalized advantages. The paper adopts the soft setting (standard GRPO loss), where shared-prefix tokens naturally receive identical importance-sampling weights $r_{i,t}(\theta)$ due to $y_{i,<t} = y_{j,<t}$, yielding more stable reward dynamics in practice.

A hierarchical reward $R$ includes correctness, format compliance, and a bonus $r_M$ for correctly invoking multiple tools (search + Python interpreter) jointly.

The theoretical grounding derives a Generalized Policy Gradient (GPG) Theorem that treats contiguous token segments between tool calls as macro-actions $MA_i$, subsuming the standard token-level policy gradient as a special case.

Key Contributions¶

First empirical quantification of token-level entropy dynamics in LLM tool-use agents, showing entropy spikes post-tool-call exceed initial-input entropy and vary by tool type (search > code interpreter).
ARPO algorithm with entropy-based adaptive rollout (global + partial branching) and advantage attribution estimation for step-level credit assignment.
GPG Theorem providing theoretical justification for macro-action-based policy gradient in Transformer agents.
4% average accuracy gain over trajectory-level RL baselines across 10 benchmarks; 6% gain on GAIA and WebwalkerQA deep search; achieves these results with 50% fewer tool calls during training.

Results¶

Math & Knowledge Reasoning (10 datasets, Llama3.1-8B): ARPO avg 55.3% vs. DAPO 50.4%, REINFORCE++ 51.1%, GRPO 51.1%. Gains of +10% on AIME24 (23.3% vs. 16.7% DAPO) and +8.4% on HotpotQA (65.4% vs. 56.6% DAPO).
Math & Knowledge Reasoning (Qwen2.5-7B): ARPO avg 58.3% vs. GRPO 56.5%, DAPO 54.8%; AIME24: 30.0% vs. 23.3% (GRPO).
Deep Search (Qwen3-14B, 1K training samples): ARPO 43.2% on GAIA and 10.0% on HLE pass@1, versus GPT-4o (2.0% HLE) and DeepSeek-R1-671B (8.6% HLE). GRPO baseline scores 6% lower on GAIA.
Pass@K scaling (Qwen3-14B): ARPO achieves 61.2% Pass@5 on GAIA, 24.0% on HLE, 59.0% on xBench-DR.
Tool-call efficiency: ARPO reaches competitive performance using ~50% of the tool-call budget required by GRPO/DAPO during RL training.

Limitations¶

Entropy thresholds ($\alpha$, $\beta$, $\tau$, $Z$, $k$) introduce multiple hyperparameters whose sensitivity is not fully ablated in the excerpt.
Evaluated with a fixed, relatively small backbone (7B–14B); scalability to much larger models is unverified.
Training data for deep search experiments is limited to 1K samples from a single open-source web search dataset, so generalization to diverse domain-specific corpora is unclear.
The branching mechanism increases rollout complexity beyond $O(n \log n)$ in worst case (up to $O(n^2)$), and the actual wall-clock overhead is not fully characterized.
Soft advantage estimation is adopted as default partly based on empirical reward stability, with the theoretical equivalence to hard estimation relegated to an appendix.

Relevance to Agentic AI / LLM Agents¶

ARPO directly targets the core training challenge of multi-turn LLM agents: how to propagate learning signal through extended tool-call loops rather than just at trajectory endpoints. The entropy-based branching criterion provides a principled, model-intrinsic signal (no auxiliary reward model needed) for deciding when to invest extra rollout computation, which is directly applicable to any tool-augmented agent training setup. The GPG Theorem formalizes the agentic RL training loop as macro-action MDP optimization, giving a theoretical backbone that unifies token-level and trajectory-level RL for agents. The 50%-tool-budget result has practical significance for real-world deployments where API calls are expensive.