Skip to content

ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset

🕒 Published (v1): 2025-11-12 13:01 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToolMind is a 360k-sample synthetic tool-use dataset combining 160k graph-sampled, multi-agent-synthesized trajectories with 200k augmented open-source samples. Its distinguishing feature is a two-stage quality filter that operates at both trajectory and individual-turn granularity, removing erroneous intermediate steps before SFT. Models fine-tuned on ToolMind achieve consistent gains on BFCL-v4, τ-bench, and τ²-bench over strong baselines.

Problem

Existing tool-use datasets suffer from limited scale, absent reasoning traces, shallow multi-turn coverage, and insufficient query diversity. Critically, prior synthesis pipelines validate quality only at the full-trajectory level, allowing erroneous intermediate turns to persist in training data; these propagate during fine-tuning and degrade model performance.

Method

Graph Construction and Function-Chain Sampling. Functions collected from xlam-function-calling-60k, glaive-function-calling-v2, and ToolACE are standardized and embedded per parameter as \(v(r) = \phi(\text{DESC} \| \text{desc}(r) \| \text{TYPE} \| \text{type}(r)) \in \mathbb{R}^d\). A directed function graph \(G=(V,E)\) is built where edge \(i \to j\) exists iff \(s_{ij} = \max_{y \in Y_i, x \in X_j} \text{sim}(v(y), v(x)) > \tau\), validated by an LLM. Function chains \(W = (f_0, f_1, \ldots, f_L)\), \(L \sim \text{Uniform}\{5,\ldots,20\}\), are sampled via random walks with per-node visit caps.

Multi-Agent Trajectory Synthesis. Three LLM-instantiated agents operate in an iterative loop: a User Agent drives the conversation from a synthesized intent, an Assistant Agent calls functions or requests clarification, and a Tool Agent simulates function execution. Tools are not actually executed; responses are simulated.

Two-Stage Quality Filtering. (1) Trajectory-level: a second LLM judges goal coverage, coherence, and cross-turn relevance. (2) Turn-level: an LLM judge inspects each individual turn, masking invalid, role-inconsistent, or assumption-incorrect steps so only correct turns enter training. Loss is computed only on anchor assistant-response tokens.

Key Contributions

  • 360k-sample ToolMind dataset: 160k synthesized + 200k augmented open-source, using 20k functions across broad domains.
  • Function-graph construction via parameter-embedding cosine similarity plus LLM validation, enabling semantically grounded random-walk function-chain sampling.
  • Turn-level quality filtering on top of trajectory-level filtering, isolating and masking erroneous intermediate steps rather than discarding full trajectories.
  • Empirical validation showing large-scale, fully synthetic data can substitute for real-world interaction data in tool-use SFT.

Results

  • BFCL-v4: Qwen3-8B overall +4.69% (42.21→46.92); Qwen3-14B +5.40% (45.14→50.54), surpassing DeepSeek-V3, GPT-4o-2024-11-20, and Qwen3-235B-instruct at a fraction of the parameter count.
  • Ï„-bench: Qwen3-8B avg +10.87% (35.83→46.70); Qwen3-14B avg +14.22% (38.78→53.00).
  • τ²-bench: Qwen3-8B avg +11.77% (34.64→46.41); Qwen3-14B avg +8.43% (40.65→49.08).
  • Ablation: removing turn-level filtering (config b) drops Ï„-bench retail from 42.61 to 42.61 vs. 57.39 with full ToolMind; synthesized data alone (config a) outperforms the no-filtering variant on BFCL-v4 (46.87 vs 44.11); combined dataset beats any single component across nearly all metrics.

Limitations

  • Tools are never actually executed; the Tool Agent simulates responses, so the dataset does not capture real API failure modes, latency, or schema drift.
  • Function graph edges rely on embedding cosine similarity plus LLM validation, which may miss semantic mismatches or introduce false positives at scale.
  • Evaluation is limited to Qwen3-8B and Qwen3-14B; generalization to architectures outside the Qwen3 family is unverified.
  • Domain distribution is skewed toward data analysis and entertainment (per Figure 4), potentially underrepresenting specialized professional APIs.
  • The per-node visit cap and random-walk length range (\(L \in [5,20]\)) are fixed heuristics with no ablation justifying these choices.

Relevance to Harnesses / Meta-Harnesses

ToolMind's data synthesis pipeline is itself a meta-harness: a multi-agent orchestration layer (User/Assistant/Tool agents) wrapped around a graph-sampling engine and a two-stage LLM-judge filtering harness, all operating without human-in-the-loop. The turn-level filtering stage is architecturally a harness that post-processes agent outputs, making it directly relevant to anyone building pipelines where intermediate LLM actions must be validated before being consumed by downstream training or evaluation systems. The ablation demonstrating the necessity of turn-level filtering quantifies a specific harness design decision—coarse trajectory validation is insufficient—which is a transferable result for anyone designing quality-gating stages in agentic data pipelines.