AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL with entropy constraints for automatic scaling of tool-use capabilities

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoTool addresses two failure modes in RL-based training for LLM tool use: reasoning collapse (models shrink to short trajectories under direct RL) and overthinking (distilled models spend 10× tokens on trivially simple queries). It introduces a warm-up SFT stage followed by RL with decoupled adaptive entropy constraints, achieving automatic think/no-think mode selection per query at inference time.

Problem

Direct RLVR (GRPO) training on tool-use tasks causes reasoning collapse: unlike math, response length decreases as accuracy improves, leaving complex multi-turn problems unsolved. Distillation-based SFT fixes multi-turn accuracy but applies long reasoning to all queries uniformly, incurring >10× token overhead with no accuracy gain. Static entropy constraints are a partial remedy but highly sensitive to the coefficient \(\beta\), making pre-selection impractical.

Method

Stage 1 — Warm-up SFT. A mixed dataset (PubTool, ~8.2k turns) is constructed by running pass@8 on training data with a no-think model (Qwen2.5-7B-Instruct) and a thinking model (Qwen3-32B). If the no-think model answers correctly, its short response is the label; otherwise the thinking model's long reasoning is used. A special auto-thinking template with [mode]think[/mode] / [mode]no_think[/mode] tags allows the model to select reasoning mode.

Stage 2 — Decoupled Adaptive Entropy Constraints (RLVR). The policy loss is: $\(L_p = \frac{1}{N}\sum_{i=1}^N \left[ -\min\!\left(\rho_i \hat{A}_i,\, \text{clip}(\rho_i,1{-}\epsilon,1{+}\epsilon)\hat{A}_i\right) - \beta_i H_i \right]\)$ where \(\beta_i = \beta_s \cdot m_i \cdot \mathbb{1}\{H_i \leq H_s\} + \beta_l \cdot (1-m_i) \cdot \mathbb{1}\{H_i \leq H_l\}\) decouples short (\(m_i=1\)) and long (\(m_i=0\)) trajectories. \(\beta_s\) is fixed; \(\beta_l\) is adaptive, updated by minimizing: $\(L_\beta^l = \frac{1}{\sum_j (1-m_j)} \sum_{i=1}^N (1-m_i)\cdot\beta_l\cdot(H_i - H_l)\)$ so \(\beta_l\) rises when \(H_i < H_l\) (insufficient exploration) and falls when \(H_i > H_l\).

Reward. Format reward checks mode-tag compliance. Answer reward is asymmetric: \(+1.0\) correct no-think, \(+0.5\) correct think, \(-0.5\) wrong think, \(-1.0\) wrong no-think — incentivizing short responses when accurate and long reasoning when uncertain.

Key Contributions

• Empirical diagnosis that reasoning collapse in tool-use RL is driven by entropy collapse, not data difficulty distribution.
• Decoupled entropy constraint separating short and long trajectory regularization, preventing short-trajectory dominance from suppressing exploration in long-reasoning paths.
• Adaptive \(\beta_l\) that eliminates manual entropy coefficient tuning.
• PubTool: curated 8.2k-SFT / 7k-RL dataset from ToolACE, xLAM, and Hermes Function-Calling with difficulty-aware quality refinement (variance-based sample prioritization).
• Inference-time controllable reasoning mode via prepended special tokens.

Results

• AutoTool-7B vs. PubTool-SFT (same data): +11.95% overall accuracy on BFCL; +28.5% on Multi-Turn subset.
• vs. Base model: +16.43% overall.
• vs. PubTool-Distilled: +9.8% accuracy, ~81% fewer inference tokens.
• BFCL overall: 70.12% (auto), matching frontier models GPT-4o (70.42%) and o3 (70.32%).
• ACEBench: +6.5 vs. GRPO, +5.9 vs. Distilled.
• Thinking rate: 45% on complex queries, 0% on simple queries (in-distribution at inference time).
• Complex-query reasoning trajectories extend 5× longer than simple-query trajectories.

Limitations

• Warm-up SFT label quality depends on pass@8 from two specific models (Qwen2.5-7B-Instruct and Qwen3-32B); the procedure requires substantial inference compute to construct PubTool.
• Evaluation confined to one base model scale (~7B); scalability to larger or smaller models is not demonstrated.
• Data quality refinement (variance-based downsampling from 21k→7k) requires multi-epoch GRPO pre-training, adding training cost.
• Ablation section is incomplete in the provided text; full component contribution analysis is not available here.
• Benchmarks (BFCL, API-Bank, ACEBench) focus on function-calling accuracy; broader agentic capabilities (e.g., web navigation, code execution) are not evaluated.

Relevance to Agentic AI / LLM Agents

AutoTool directly advances a core bottleneck in deploying LLM agents: efficiently scaling reasoning depth to query complexity rather than using a fixed inference budget. The finding that entropy collapse — not data difficulty — drives RL training failure in tool-use has broad implications for any agentic RLVR recipe. The asymmetric reward design (penalizing unnecessary thinking more than unnecessary brevity) is a transferable principle for multi-step agentic workflows where token cost compounds across turns. This work also provides a practical recipe for test-time adaptive compute in tool-calling agents, complementing emerging work on mixture-of-depth and early-exit inference strategies.