Skip to content

Benchmarking LLM Tool-Use in the Wild

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Wild benchmark capturing messy, multi-turn, multi-step real tool-use scenarios

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WildToolBench is a multi-turn, multi-step LLM tool-use benchmark grounded in real user behavior that exposes three "in-the-wild" challenges absent from prior benchmarks: compositional task orchestration, implicit intent inference across dialogue turns, and policy switching under instruction transition. Evaluating 57 LLMs reveals that no model exceeds 15% session accuracy, demonstrating that existing benchmarks overstate agentic progress. The benchmark comprises 256 scenarios and 1,024 tasks derived from real user logs with human-in-the-loop annotation.

Problem

Existing tool-use benchmarks (BFCL, Ï„-Bench, ToolBench, etc.) are saturated and unrealistically idealized: tasks are syntactically independent and identically distributed, user intent is fully provided, and instruction types do not mix. LLM-as-User simulators diverge from real behavior (too flawless, unstable). This renders apparent SOTA progress spurious and fails to identify the true bottlenecks of agentic tool-use.

Method

Data pipeline. Real user logs are analyzed to extract 256 seed scenarios covering 8 tool categories. From 1,600+ publicly available APIs, per-scenario tool subsets are selected. GPT-4o drives a multi-agent (user + assistant) simulation to generate initial trajectories; each tool invocation is then manually verified and annotated as ground truth, producing 1,024 tasks (4 per scenario).

Three challenge axes deliberately embedded: 1. Compositional tasks (Challenge 1): multiple simple sub-requirements in one instruction require planning a tool-calling topological graph (possibly a tree with parallel branches) rather than a chain. 2. Hidden intention (Challenge 2): 100% of tasks involve partial information, coreferential references, or long-range dependencies (i− j > 2 turns) that force contextual inference. 3. Instruction transition (Challenge 3): dialogues mix \(g_{\text{single}}\), \(g_{\text{multi}}\), \(g_{\text{chat}}\), \(g_{\text{clarify}}\) task types; the LLM must dynamically switch among tool-use, direct answering, and proactive clarification policies.

Evaluation metrics. Beyond task accuracy and session accuracy (all 4 tasks correct): - Accomplish Progress (AP) Rate: fraction of valid tool-call nodes in a partially completed trajectory. - Optimal Path (OP) Rate: whether the completed execution path has minimum depth among all enumerated valid decision trees, measured via depth-first topological sort + incremental path matching.

Key Contributions

  • WildToolBench dataset: 256 real-user-grounded scenarios, 1,024 tasks, 1,600+ verified APIs, human-annotated ground-truth trajectories.
  • Three-challenge taxonomy (compositionality, vagueness, variability) operationalized with quantifiable annotation criteria.
  • Enumerate-match-score evaluation pipeline for compositional tool orchestration, enumerating all valid topological paths via DFS and tracking AP/OP metrics.
  • 57-model evaluation spanning proprietary, open-source general, and open-source specialized (tool-tuned) models.
  • Empirical evidence that the bottleneck is real user behavior, not artificial task complexity.

Results

  • No model exceeds 15% session accuracy; most fall below 60% task accuracy.
  • Best proprietary models (Gemini-2.0-Thinking, Claude-4-Sonnet, GPT-4o): session accuracy ~14–15%; task accuracy ~61%.
  • Best open-source general models (GLM-4.5, Kimi-K2): session accuracy ~12%; task accuracy ~56%.
  • Tool-specialized open-source models (xLAM-2-70B, ToolACE2-8B, Watt-8B): session accuracy ~4–8%; \(g_{\text{chat}}\) accuracy high (~79–95%) but \(g_{\text{clarify}}\) and \(g_{\text{multi}}\) severely degraded.
  • Reasoning models consistently outperform non-reasoning counterparts.
  • Task accuracy degrades monotonically with task order within a session (more context dependency → worse performance).
  • \(g_{\text{clarify}}\) tasks are the hardest: LLMs frequently fire a function call instead of requesting clarification.
  • WildToolBench vs. BFCL-v3: average 5.27 turns and 1.92 tool-call steps per task (vs. 3.75 turns, 1.68 steps for BFCL).

Limitations

  • Benchmark covers 256 scenarios and 1,024 tasks; coverage of the full diversity of real-world tool-use domains remains incomplete.
  • Ground-truth annotation relies on GPT-4o-driven initial trajectories, which may introduce LLM-specific biases into the task space.
  • OP Rate evaluation requires exhaustive DFS enumeration of all valid paths, which may not scale to tasks with very large tool dependency graphs.
  • Only tool-call correctness is evaluated; broader agentic metrics (e.g., efficiency, safety, user satisfaction) are not addressed.
  • Session accuracy as a metric is strict (all 4 tasks must be correct), but its relationship to real-world utility is not empirically validated.

Relevance to Agentic AI / LLM Agents

WildToolBench directly targets the agentic loop where an LLM must plan multi-step tool calls, manage dialogue state, and adapt policies across heterogeneous user instructions—the core competency of any deployed LLM agent. The finding that no model exceeds 15% session accuracy despite high single-call accuracy is a strong empirical signal that current RLHF/instruction-tuning pipelines optimize for idealized, isolated tasks rather than the compositional, context-dependent, and dynamically shifting interactions that characterize real deployments. The AP/OP metrics for measuring tool-call topology quality are directly reusable in agent evaluation frameworks and planning research. The three-challenge taxonomy (compositionality, vagueness, variability) provides a principled axis for future agent training data curation and reward shaping.