Skip to content

ToolRL: Reward is All Tool Learning Needs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToolRL is a systematic study on reward design for training LLMs to use tools via reinforcement learning (RL), specifically under GRPO and PPO. It proposes a principled, fine-grained reward decomposing tool call correctness into tool name, parameter name, and parameter value matching, with dynamic scaling over training. The approach outperforms SFT baselines by 15% and base models by 17% across tool-use and QA benchmarks.

Problem

SFT for Tool-Integrated Reasoning (TIR) overfits to training trajectories and fails to generalize to novel tool configurations or task types. Existing RL approaches for tool use (e.g., Search-R1, TORL) use coarse-grained rewards (binary answer matching) and are restricted to narrow tool domains (search or code). No systematic analysis existed for reward design in general-purpose, multi-tool, multi-parameter RL training.

Method

At each reasoning step, the model generates a <think> trace, zero or more <tool_call> invocations, and a <response>. The reward has two additive components:

  • Format reward (binary): checks all required special tokens appear in correct order.
  • Correctness reward (continuous, ∈ [−R_max, R_max]): an optimal matching score r_match = r_name + r_param + r_value, where each sub-score is a Jaccard-style set overlap between predicted and ground-truth tool names, parameter key sets, and parameter values. The total is normalized to [−3, 3] (R_max=3 empirically).

Dynamic reward scaling smoothly interpolates emphasis from format-fidelity to correctness as training progresses. KL regularization is omitted to allow freer exploration and faster convergence. Training uses GRPO (primary) and PPO on Qwen2.5-Instruct (1.5B/3B/7B) and Llama-3.2-3B, with 4K mixed examples from ToolACE, Hammer (masked), and xLAM, for 15 epochs via the veRL framework.

Key Contributions

  • First systematic study of reward design for RL-based general-purpose tool selection and application in LLMs.
  • Fine-grained correctness reward decomposed into tool name, parameter name, and parameter value matching with continuous partial credit.
  • Empirical analysis across four reward dimensions: type, scale, granularity, and temporal dynamics.
  • Evidence that GRPO cold-start outperforms SFT-initialized RL, suggesting SFT pretraining induces overfitting that limits subsequent RL generalization.
  • Demonstration of emergent behaviors: proactive tool avoidance, metacognitive reasoning, and irrelevant tool rejection—none explicitly trained.

Results

  • BFCL V3 (overall accuracy): GRPO cold-start achieves 46.20% (1.5B), 52.98% (3B), 58.38% (7B) vs. raw instruct baselines of 19.41%, 33.04%, 41.97%; SFT4k baselines of 40.67%, 41.97%, 36.53%.
  • API-Bank: Qwen2.5-3B GRPO cold-start reaches 67.00% overall vs. 51.59% raw, 50.92% SFT4k.
  • Bamboogle (QA generalization): Qwen2.5-7B GRPO cold-start achieves 72.0% accuracy vs. 40.0% Search-R1 GRPO, 69.6% raw model—with efficient tool call usage (avg 1.63 calls vs. 3.54–3.70 for SFT variants).
  • Data scaling ablation: Scaling from 4K→10K RL examples yields <0.5% gain, demonstrating reward quality dominates data quantity.
  • Reward scale ablation: Correctness scale=3 vs. scale=1 improves BFCL from 40.62% to 52.98%.
  • Granularity ablation: Coarse (whole-call binary) reward degrades performance relative to fine-grained decomposition by up to ~10% absolute (Qwen2.5-1.5B).
  • Length reward: Both static and dynamic length rewards degrade or do not improve performance, with Qwen2.5-1.5B dropping from 46.20% to 28.51% under dynamic length reward.

Limitations

  • Evaluation capped at 7B parameter models; scalability to larger or smaller models is unexplored.
  • Only policy-gradient methods (GRPO, PPO) evaluated; applicability to preference-based methods (DPO, SimPO) is untested.
  • Training data limited to three tool-use datasets (4K examples total); broader distribution coverage not assessed.
  • Llama-3.2 models show weaker gains than Qwen2.5, suggesting architecture-dependent sensitivity to GRPO reward shaping.
  • Ground-truth tool call labels required at each step for step-wise reward; deployment in settings lacking oracle intermediate annotations is not addressed.

Relevance to Agentic AI / LLM Agents

ToolRL directly addresses a core bottleneck in building capable LLM agents: how to train reliable, generalizable tool-use behavior without costly trajectory-level supervision. The fine-grained GRPO reward framework yields emergent proactiveness and metacognition—agents that voluntarily avoid tool invocation when it's unnecessary, a prerequisite for safe deployment. The finding that SFT initialization hurts RL generalization challenges the common practice of SFT-then-RL pipelines for agent training. The demonstrated transfer to free-form QA (Bamboogle) without retraining supports the claim that reward shaping, not data scale, drives generalization—critical guidance for practitioners designing training pipelines for general-purpose agents.