Skip to content

AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

🕒 Published (v1): 2025-12-15 12:38 UTC · Source: Arxiv · Venue: ICCV 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoTool is a training framework that equips LLM agents with dynamic tool-selection across evolving toolsets, as opposed to the standard assumption of a fixed, predefined tool inventory. It employs a dual-phase pipeline—SFT+RL trajectory stabilization followed by KL-regularized Plackett–Luce (PL) ranking focused on tool-selection steps—and trains on a 200k dataset spanning 1,346 tools. On ten benchmarks, AutoTool (Qwen3-8B) outperforms fixed-tool agentic RL baselines by 6.4% on math/science, 7.7% on code, and generalizes to tools entirely unseen during training.

Problem

Existing agentic RL methods assume a closed, domain-specific tool inventory: the policy learns to invoke a predefined set of tools, but cannot adapt when new tools are introduced at inference time. This causes overfitting to a static toolset and failure in realistic evolving environments where the agent must (i) select from a diverse multi-domain toolset, and (ii) use tools that were never seen during training.

Method

Dual-phase optimization pipeline:

  • Phase I – Trajectory Stabilization: Fine-tune a base reasoning model with SFT on curated trajectories, then apply RL policy optimization (GRPO-style) to stabilize interleaved CoT reasoning, tool-selection, and tool-integration steps.
  • Phase II – Tool-Selection Refinement: Freeze reasoning/integration steps and optimize only tool-selection segments. Each tool t_k is embedded via the LLM's own embedding layer as e_{t_k} = Emb_πθ[t_k, μ(t_k)]. At selection step Ï„_i^select, the agent generates a rationale and an anchor embedding e'_i; the tool is chosen by softmax over negative squared Frobenius distances between e'_i and all candidate tool embeddings, enabling generalization to unseen tools via representation alignment.
  • PL Ranking: N trajectory rollouts per query are scored by a step-level reward combining a Process Reward Model (PRM) over the rationale and answer accuracy (r_tool = PRM(Ï„_i^select) + Acc(x, t_k)). Trajectories are ranked under the Plackett–Luce model; the optimal policy is shown to be equivalent to matching the PL ranking distribution. The tractable training objective is a KL-regularized cross-entropy loss over reward-ranked rollouts.
  • Dataset: A 200k-trajectory dataset is curated via three stages: (1) assembling 1,346 tools across code, search, and multimodal categories and 120 task types; (2) using an expert model to generate explicit tool-selection rationales before each tool invocation; (3) trajectory augmentation and LLM-as-judge filtering. Only 460 tools (34.2%) appear during training; 886 (65.8%) are held out for inference-time evaluation.

Key Contributions

  • Dual-phase agentic training pipeline that explicitly separates trajectory stabilization (Phase I) from tool-selection refinement (Phase II)
  • Embedding-anchored tool-selection mechanism that operates over frozen tool embeddings via vector distance, enabling zero-shot generalization to unseen tools
  • Theoretical proof of equivalence between the KL-regularized RL optimal policy and the PL ranking distribution over tool-scored rollouts (Theorem 2.1)
  • 200k-scale dynamic tool-use dataset with tool-selection rationales covering 1,346 tools and 120 tasks across math, science, code, search, and multimodal domains
  • Demonstration that Phase II PL ranking yields consistent gains over Phase I–only SFT/GRPO/ARPO baselines across all domains

Results

  • vs. agentic training baselines (Qwen3-8B): AutoTool vs. ARPO — AIME24: 68.8% vs. 56.7% (+12.1%), AIME25: 53.3% vs. 46.7%, HotpotQA: 45.1% vs. 38.6%, 2Wiki: 48.8% vs. 45.2%, V-Code: 56.1% vs. 55.2%
  • vs. advanced LLM agents: AutoTool (Qwen3-8B, 8B) outperforms QwQ-32B (32B) on AIME24 (68.8% vs. 22.3%) and all Qwen2.5-Math/VL-72B variants on math benchmarks
  • vs. fixed tool-integration specialists: AutoTool surpasses ReTool (32B) on HotpotQA (45.1% vs. 21.4%) and Search-R1 on AIME24 (68.8% vs. 12.1%)
  • Unseen-tool out-of-domain (Qwen3-8B, only unseen tools): AIME24 66.7%, HotpotQA 43.8%, LiveCodeBench-V2 77.9%, MedQA 84.3% — outperforming both Search-R1 and ReTool across all four
  • Restricting inference to the 886 unseen tools causes <2% drop vs. full toolset
  • Tool-selection rationales increase successful tool execution rate by 13.5%

Limitations

  • Only evaluated on two base models (Qwen3-8B, Qwen2.5-VL-7B); generalization to architectures with different embedding spaces is unverified
  • Embedding-anchored selection requires all tools to be encodable in the LLM's own embedding space; tools with heterogeneous or non-textual representations are not addressed
  • Dataset curation requires an expert reasoning model for rationale generation, which introduces cost and potential annotation noise not fully characterized
  • Phase II PL ranking with N=8 rollouts per query substantially increases training compute relative to standard GRPO
  • No ablation on the sensitivity to PRM quality; a weak PRM could degrade the step-level reward signal

Relevance to Agentic AI / LLM Agents

AutoTool directly attacks the closed-tool-inventory assumption that limits virtually all current agentic RL work, making it highly relevant to building agents deployable in open, evolving environments. The embedding-anchored selection and PL ranking formulation are architecture-level contributions that could be adopted in future agentic frameworks beyond the specific benchmarks tested. The explicit separation of reasoning stabilization from tool-selection optimization is a reusable design pattern for multi-phase agentic training pipelines. The 200k open dataset spanning 1,346 tools fills a significant gap in training data for dynamic tool-use research.