Skip to content

TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Trajectory-aware evaluation of tool use beyond just final answer accuracy

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TRAJECT-Bench is a trajectory-aware benchmark evaluating LLM tool use across 1,228 executable production-style APIs and 5,670 queries spanning parallel and sequential tool-call structures. Unlike prior benchmarks that grade only final answers, it measures trajectory-level correctness: tool selection, argument validity, and dependency ordering. Evaluation of frontier models reveals that the sharpest performance drop occurs in transitioning from short (3-tool) to mid-length (5-tool) trajectories, not from mid to long.

Problem

Existing tool-use benchmarks (ToolBench, BFCL, Gorilla, ToolQA) privilege end-task accuracy and overlook intermediate trajectory quality—whether the correct tools are chosen in the right order with correct parameters. They also use small/simulated tool sets and simple, explicit queries that spell out API names, diverging from real agentic settings where queries are indirect and tool sets are large.

Method

Tool set. 1,228 tools curated from RapidAPI across 10 domains; validated for executability, deduped for minimal overlap, enriched with empirical I/O descriptions, and filtered for parameter richness.

Trajectory synthesis (two structures): - Parallel: an LLM generates unordered, independent tool-call sets (3–10+ calls) from domain task-type templates; each call has fully specified, non-interdependent inputs. - Sequential: a directed tool graph \(G_T=(V,E)\) encodes output→input bindings between tools; human-designed sequence templates are instantiated by LLM with concrete parameters, producing dependency chains of variable depth \(n_\text{traj}\).

Query difficulty. Each trajectory is paired with a simple (explicit, parameter-specifying) and a hard (indirect, colloquial) query variant.

Evaluation metrics: - EM — predicted tool names exactly match ground-truth trajectory - Inclusion — fraction of ground-truth tools present in prediction - Tool Usage (Usage) — schema, format, and value correctness of predicted parameters - Traj-Satisfy — LLM-judge score assessing whether the predicted trajectory would solve the user query (used when gold traces are unavailable) - Acc — LLM-judge final-answer correctness

Tool selection strategies compared: full-context (all), domain-filtered (domain), and retrieval-based (all-MiniLM-L6-v2, bge-large-en-v1.5, ToolBench-IR).

Key Contributions

  • First benchmark combining practical executable tools (1,228 APIs), multi-structure trajectories (parallel + sequential), variable trajectory depth (3–10+ tools), two query difficulty tiers, and trajectory-aware metrics—all simultaneously.
  • Identification of three recurring failure modes: similar-tool confusion, parameter-blind selection, and redundant tool calling.
  • Empirical finding that the primary scaling bottleneck is the short-to-mid transition (3→5 tools), not the mid-to-long segment.
  • Evaluation of both static LLM prompting and agentic inference (ReAct with static/dynamic retrieval) alongside trained tool-calling APIs.

Results

  • Top models: Gemini-2.5-pro leads on simple queries; Claude-4 matches it on hard versions.
  • Simple vs. hard gap: Claude-4 EM drops 0.846→0.445; Gemini-2.5-pro drops 0.851→0.442 under direct prompting.
  • Parallel vs. sequential: Gemini-2.5-pro EM/Inclusion/Usage on simple parallel (0.851/0.854/0.835) exceeds sequential (0.807/0.821/0.809).
  • Inclusion–EM gap on hard queries: Claude-3.7 0.135 EM vs. 0.554 Inclusion; Gemini-2.5-flash 0.216 EM vs. 0.538 Inclusion—models retrieve most tools but miss exact sets.
  • Scaling: all models degrade as trajectory length grows; sharpest drop at 3→5 tools. Claude-4 and DeepSeek-V3.1 are most robust at long chains; o4-mini and Kimi-k2 collapse sharply beyond 7 tools.
  • LLM-judge proxy validity: Traj-Satisfy tracks EM closely (Claude-4: 8.549 judge ↔ 0.846 EM on simple; 4.882 ↔ 0.445 on hard).
  • Retrieval: retrieval-based selection underperforms domain-filtered, indicating current embedding retrievers miss tools needed for indirect queries.

Limitations

  • Trajectory structures restricted to parallel and chain; richer DAG topologies are left to future work.
  • Ten domains covered; coverage is not exhaustive.
  • Sequential templates are manually designed, limiting scalability of that split.
  • Retrieval evaluation uses fixed top-20 retrieved tools; optimal \(k\) is not explored.
  • Agentic evaluation focuses solely on ReAct; other agent architectures (e.g., plan-and-execute, tree-search) are deferred to appendices/future work.
  • Ground-truth trajectories are synthetically generated, potentially missing the full diversity of real user intents.

Relevance to Agentic AI / LLM Agents

TRAJECT-Bench directly targets the core bottleneck of LLM agents in production: reliable multi-step tool orchestration with large, heterogeneous APIs. The trajectory-aware metrics expose failure modes—similar-tool confusion, parameter blindness, redundant calling—that final-answer metrics systematically hide, making it actionable for training and inference improvements. The discovery that the short-to-mid-length transition is the primary scaling bottleneck reshapes where effort in tool-use training should be concentrated. The agentic evaluation layer (ReAct with dynamic retrieval) extends the benchmark beyond single-turn function-calling into the sequential decision-making regime central to real deployed agents.