OrchestrationBench: LLM-Driven Agentic Planning and Tool Use in Multi-Domain Scenarios¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-domain agentic planning and tool-use benchmark; comprehensive orchestration evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

OrchestrationBench is a bilingual (English/Korean) benchmark evaluating LLMs on two disentangled dimensions: DAG-based workflow planning and constraint-aware tool execution across 17 domains with ~100 virtual tools. It targets the "LLM-to-LLM" orchestration setting where a main LLM decomposes tasks and coordinates specialist sub-LLMs. Experiments on state-of-the-art models reveal that function-calling performance is comparatively consistent across models while planning ability varies substantially.

Problem¶

Existing benchmarks assess either single-agent tool use or isolated planning in simplified, domain-specific settings. They conflate planning errors with execution errors, lack multi-agent (LLM-to-LLM) orchestration scenarios, omit realistic business constraints (e.g., constraint validation, user clarification, dynamic workflow revision), and are largely English-only — leaving orchestration capability for commercial, multilingual service deployments under-evaluated.

Method¶

OrchestrationBench formalizes orchestration as a two-phase evaluation:

Workflow Planning: User requests are decomposed into a Directed Acyclic Graph (DAG) encoded as a YAML workflow schema with fields for execution state (status), dependency type (independent/dependent), prerequisite workflows (depend_on), step-level sub-LLM assignment (name), and refined queries. Planning quality is measured by \(1 - \text{GED}\) (normalized Graph Edit Distance), reported hierarchically: a structural score (workflow topology) and a component score (step assignments), with selection errors weighted \(0.8\) vs. status errors \(0.2\).
Tool Execution: Sub-LLMs process refined queries and are evaluated on (a) call/reject classification accuracy — distinguishing valid invocations from AWAIT_FOR_USER_INPUT and TOOL_CONSTRAINT_VIOLATION rejections — and (b) function-calling performance on correctly invoked tools (tool selection, argument extraction, value validation). LLM-based auxiliary judges handle semantic correctness.

The dataset (219 English sessions, 222 Korean sessions; 317/324 planning cases; 706/730 tool call instances) was constructed entirely through manual annotation by trained annotators, cross-validated by ≥3 independent reviewers, with no synthetic generation to avoid model-specific bias. Bilingual expansion from Korean to English preserves cultural specificity. Each scenario is run 3× at temperature 0.2.

Key Contributions¶

First benchmark explicitly disentangling workflow-level DAG planning from constraint-aware tool execution in a multi-agent (LLM-to-LLM) orchestration setting.
Bilingual (EN/KO) manually annotated dataset spanning 17 domains, ~100 virtual tools, covering sequential/parallel workflows, multi-domain tasks, constraint violations, user clarifications, and dynamic mid-session revisions.
Structured planning metric based on normalized \(1-\text{GED}\) with hierarchical structural and component scoring.
Tool execution protocol separating call/reject classification from function-calling accuracy, enabling precise failure diagnosis.
Living benchmark architecture, extensible to new domains, tools, and languages without retraining or API dependencies.

Results¶

(Paper text truncated before full results tables; only headline findings are reported.) - Function-calling performance is relatively consistent across evaluated models (gpt-4.1, gpt-4o, gpt-5, claude-sonnet-4, gemini-2.5-pro/flash-preview, Qwen3 series, A.X-4.0, kanana-1.5, EXAONE-4.0). - Planning capabilities show substantial variation across models, with no model achieving uniformly strong performance on DAG-structured workflow generation. - State-of-the-art agents achieve less than 50% success on dynamic multi-turn interactions in comparable prior benchmarks (τ-bench, GAIA, UltraTool), motivating structured orchestration evaluation.

Limitations¶

Selection error weight (\(0.8\) vs. \(0.2\) for status errors) in \(1-\text{GED}\) is not empirically derived, introducing potential metric bias.
Dataset scale is modest (~220 sessions per language), which may limit statistical robustness for fine-grained per-domain analysis.
Virtual tools are domain-generalized and do not connect to live APIs, limiting direct transferability to production deployment assessment.
Benchmark covers primarily everyday consumer services; enterprise/utility workflows are underrepresented.
Cultural adaptation between EN and KO introduces slight tool count discrepancies (97 vs. 99 tools), complicating direct cross-lingual comparison.

Relevance to Agentic AI / LLM Agents¶

OrchestrationBench directly addresses a critical gap in evaluating LLM agents in realistic multi-agent, multi-domain pipelines — specifically the "LLM-to-LLM" orchestration paradigm where a main model coordinates specialist sub-agents, which maps onto emerging production agent architectures. The explicit disentanglement of planning (DAG construction) from execution (tool invocation + constraint validation) provides a diagnostic lens that single-agent benchmarks cannot offer, making it valuable for identifying where agent systems break down. The finding that planning is the high-variance bottleneck — not tool calling — has direct implications for prioritizing planning-focused improvements in agentic frameworks. The bilingual design and business-constraint coverage extend evaluation to deployment-realistic scenarios beyond academic toy settings.