Skip to content

ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Yujia Qin (MSRA/Tsinghua); RL-trained strategic tool selection, strong benchmark results

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReTool trains LLMs to strategically invoke a code interpreter (CI) during long-chain reasoning via reinforcement learning, using outcome-based rewards rather than imitation learning. A 32B model achieves 67% on AIME2024 in 400 training steps, versus 40% for text-only RL at 1080 steps, and exhibits emergent behaviors including code self-correction and adaptive tool selection.

Problem

Reasoning LLMs trained with RL excel at textual chain-of-thought but fail at tasks requiring precise numerical computation, symbolic manipulation, or geometric reasoning—domains where code interpreters have a natural advantage. Prior tool-augmented approaches (SFT/prompting) merely imitate curated data distributions and cannot adaptively decide when and how to invoke tools in novel contexts.

Method

ReTool proceeds in two stages:

  1. Cold-start SFT: A pipeline converts existing text-based mathematical reasoning traces (\(D_{init}\)) into code-augmented traces (\(D_{CI}\)) by replacing manual computation steps with executable code snippets plus interpreter outputs, verified via format and answer checks. The model is fine-tuned on \(D_{CI}\) for initial tool-use competency.

  2. CI-integrated RL: PPO is applied with the modified objective: $\(J_{PPO}(\theta) = \mathbb{E}\left[\min\left(\frac{\pi_\theta(o_t|q,o_{<t};\text{CI})}{\pi_{\theta_{old}}(o_t|q,o_{<t};\text{CI})}\hat{A}_t,\;\text{clip}(\cdot, 1-\varepsilon, 1+\varepsilon)\hat{A}_t\right)\right]\)$ During rollout, the policy interleaves text and <code>...</code> blocks; a sandboxed interpreter executes each code block and returns output (or error messages) wrapped in <interpreter>...</interpreter> tags, producing hybrid trajectories \([t_1 \oplus c_1 \oplus f_1 \oplus \ldots \oplus o]\). A binary \(\{+1, -1\}\) accuracy reward on final answers guides learning. Interpreter feedback tokens are masked from loss computation; KV-cache is reused across code execution pauses for efficiency.

Key Contributions

  • Novel RL framework where rollouts include multi-turn real-time code execution inside a sandboxed CI, with outcome-only rewards (no executability reward) to prevent reward hacking.
  • Cold-start data construction pipeline converting text reasoning traces to code-augmented traces via LLM transformation + dual-stage verification.
  • Asynchronous, load-balanced sandbox pool for scalable parallel RL training.
  • Comprehensive behavioral analysis documenting CI-related metric evolution: code ratio, code lines, invocation timing, self-correction frequency, and code purpose diversity.
  • Demonstration of generalization to web search (Bing via MCP) without domain-specific modifications.

Results

  • AIME2024: ReTool (Qwen2.5-32B-Instruct) 67.0% vs. text-based RL 40.0% (400 vs. 1080 steps); ReTool (DeepSeek-R1-Distill-Qwen-32B) 72.5%, surpassing OpenAI o1-preview (44.6%) by +27.9%.
  • AIME2025: ReTool (Qwen2.5-32B) 49.3% vs. o1-preview 37.9% (+11.4%); DeepSeek backbone: 54.3%.
  • MATH500: 93.1% / 95.2%; GSM8K: 95.9% / 96.3%; GPQA: 58.7% / 62.3%—exceeding Qwen2.5-Math-72B on all benchmarks despite smaller parameter count.
  • Web search (GAIA, BrowseComp-ZH): Outperforms WebDancer and Search-o1 on same backbone.
  • Ablation: Removing RL drops AIME2024 to 40.9%; removing CI (text RL) drops to 36.7%; base model alone: 26.7%.
  • Efficiency: Final average response length ~40% shorter after RL (10k→6k tokens); code lines grow ~5× during training; code ratio reaches ~98%.

Limitations

  • Evaluated primarily on mathematical olympiad benchmarks (AIME); generalization to broader agentic tasks (multi-step planning, real-world tool APIs) is only briefly demonstrated via web search.
  • Binary \(\{+1, -1\}\) outcome reward requires ground-truth answers; inapplicable to open-ended generation tasks.
  • Code interpreter assumed reliable and sandboxed; robustness to interpreter failures or adversarial inputs not analyzed.
  • Behavioral metrics (self-correction detection via keyword heuristics) are approximate proxies.
  • Backbone-dependent: gains build on strong base models (Qwen2.5-32B, DeepSeek-R1-Distill).

Relevance to Agentic AI / LLM Agents

ReTool directly advances the core agentic capability of tool use: rather than hand-crafting tool-invocation heuristics, it uses RL to let the model self-discover when, how often, and at what point in a reasoning chain to invoke external tools—a fundamental building block for LLM agents that must coordinate with environments. The emergent code self-correction behavior demonstrates that metacognitive tool management (detecting failures and recovering) can arise from outcome-only rewards, without explicit supervision, which is a key property for robust autonomous agents. The generalization to web search via MCP tool definitions suggests the framework is architecturally compatible with the broader MCP/tool-call paradigm used in modern agentic pipelines. This work sits at the intersection of test-time compute scaling and tool-augmented reasoning, informing how future agents might allocate "thinking budget" across symbolic and neural computation.