Agentic RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper trains base LLMs to spontaneously learn Python code execution for math via outcome-based RL (ZeroTIR), without any supervised tool-use examples. It identifies an "Agent RL Scaling Law": code execution frequency, response length, and task accuracy all scale predictably with training steps. A 7B ZTRL model achieves 54.0% average on AIME24/25 + MATH500, outperforming both non-tool ZeroRL baselines and specialized math-instruct models.

Problem

Existing Tool-Integrated Reasoning (TIR) methods either rely on SFT with curated tool-use trajectories or predefined prompt triggers to invoke code interpreters, preventing genuine autonomous tool discovery. RL from fine-tuned models with existing tooling capability obscures the relationship between training dynamics and tool acquisition. The paper asks whether a base LLM can spontaneously learn when and how to use code execution purely from outcome rewards, and whether this learning follows quantifiable scaling patterns.

Method

ZeroTIR (Zero Tool-Integrated Reasoning): Base LLMs (Qwen 2.5, 1.5B/7B/32B) are trained from scratch via PPO or Reinforce++ with only outcome-based correctness rewards—no tool-use demonstrations. A decoupled Python code execution server handles tool calls via HTTP, keeping it independent of the RL training loop. During rollout, a state-machine using dynamic stop tokens (```python, ```) iteratively manages reasoning, code generation, execution, and result integration without post-hoc parsing. Key training stability techniques: (1) replay buffer filtering that drops groups with accuracy >0.8 or <0.2, focusing gradients on intermediate difficulty; (2) a maximum call limit Nmax per trajectory. For efficiency, an asynchronous pipelining framework within OpenRLHF overlaps rollout generation with parameter updates, achieving ~4× speedup over synchronous baselines. The RL objective maximizes expected outcome reward regularized by KL divergence from the reference policy. For PPO, value loss and GAE advantage estimation mask environment-inserted tokens (execution outputs) to prevent spurious updates.

Key Contributions

• Identification and empirical characterization of Agent RL Scaling Law: strong positive correlations (Spearman ρ = 0.686, log-log R² = 0.93) between training steps and code usage frequency, response length, and task accuracy—all emerging from base models with zero tool-use supervision.
• ARL framework: decoupled code execution environment + dynamic stop-token rollout mechanism compatible with OpenRLHF and Open-Reasoner-Zero, supporting PPO and Reinforce++.
• Demonstration that ZTRL (Qwen 2.5 Base 7B) achieves 54.0% avg on AIME24/25+MATH500, outperforming TORL (51.8%, math-specialized base) and all non-tool ZeroRL baselines.
• Analysis of training-inference interaction budget alignment: models trained with higher Nmax benefit more from larger inference budgets, but may underperform low-budget models in one-shot regimes.
• Finding that despite larger call budgets, agents converge to single-call strategies: >90% of correct tool-using responses use exactly one code execution.

Results

• ZTRL 7B: AIME24 46.7%, AIME25 30.0%, MATH500 85.2%, Avg 54.0%, code usage 90% — vs. TORL 7B (math-specialized base): 43.3% / 30.0% / 82.2% / 51.8% / 83%
• ZTRL 7B outperforms SimpleRL-Zero (39.1%), Eurus-2-PRIME (39.7%), Qwen2.5-Math-Instruct with TIR (41.6%), and Qwen2.5-Instruct (36.4%)
• ZTRL 32B (Reinforce++, DeepMath, Nmax=4): AIME25 56.7%, AIME24 33.3%, MATH500 89.4%, Avg 38.2%
• Interaction cap ablation (7B PPO, orz-57k): Nmax=0→20 improves avg from 10.6% to 17.2% on 1.5B; gains taper beyond Nmax=4
• Reinforce++ converges ~300 steps faster than PPO to comparable final accuracy at 7B scale
• Asynchronous pipeline: ~1.6× faster than basic async rollout, ~4× faster than synchronous server
• Log-log fit of effective training steps vs. accuracy: R² = 0.93

Limitations

• The scaling law is characterized qualitatively; rigorous derivation of the precise mathematical form (exponents, functional form) is left to future work.
• Experiments are constrained by compute: interaction counts capped at Nmax=20; very large-scale runs (e.g., unconstrained multi-call 32B training) not fully explored.
• Evaluated only on mathematical reasoning; generalization to other agentic domains (web search, multi-tool) is not demonstrated.
• Training on general Qwen 2.5 Base rather than domain-specialized math base may limit direct ceiling comparisons with models like Qwen 2.5 Math Base.
• The replay buffer filtering heuristic (0.2–0.8 accuracy band) introduces a hyperparameter not systematically ablated.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core question in agentic LLM research: can agents learn to autonomously invoke external tools without explicit supervision? The discovery that code-execution frequency, response length, and accuracy all co-scale predictably with RL training steps provides the first empirical "Agent RL Scaling Law," giving the community a quantifiable handle on how agentic tool-use capability emerges. The finding that agents converge to sparse, single-call strategies despite larger budgets has practical implications for inference-time compute allocation in deployed agentic systems. The decoupled, network-accessible execution environment architecture is directly applicable to any Agent RL setup requiring reliable external tool integration at scale.