DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows

🕒 Published (v1): 2026-05-18 20:37 UTC · Source: Arxiv · link

Why this paper was selected

DecisionBench: benchmarks emergent delegation across 11 models in long-horizon workflows

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DecisionBench is a fixed benchmark substrate for measuring emergent delegation — whether an LLM orchestrator autonomously routes sub-tasks to peer models — across GAIA, τ-bench, and BFCL multi-turn. A five-condition reference sweep over 11 models (n=23,375 tasks) reveals that end-task quality is statistically flat across all peer-awareness conditions, while routing fidelity varies by up to 4×, making process-level metrics essential. A counterfactual ceiling shows perfect single-step delegation would exceed measured performance by 15–31 pp, quantifying unrealized headroom.

Problem

Existing agentic benchmarks measure single-agent capability on fixed tasks; multi-agent systems research uses hand-coded role assignment; cost-aware routing treats delegation as a learned external policy. No prior benchmark isolates and measures emergent delegation behavior — whether an orchestrator, given a call_model tool and optional peer descriptions, delegates well during long-horizon task execution — or provides process-level metrics to diagnose how those decisions are made.

Method

DecisionBench fixes five substrate components: (1) Task suite — GAIA, τ-bench, and BFCL multi-turn, each split 20/80 into Stage-1 (profiling) and Stage-2 (evaluation) splits; (2) Peer-model pool — 11 models across 7 vendor families, pinned to a 2026-04-29 freeze date via OpenRouter; (3) Delegation interface — a call_model(name, subtask, budget_usd) tool plus an optional read_profile(model) channel for peer-information interventions; (4) Annotation layer — a frozen 7-skill taxonomy (tool-call schema adherence, multi-turn state tracking, domain-policy compliance, information retrieval, multi-step reasoning, numerical computation, long-input handling) with a deterministic rule-based step tagger (no LLM judgment) that assigns one skill per trajectory step from trace-only signals; (5) Metric suite — quality/cost/latency, quality-cost Pareto hypervolume, delegation rate, routing fidelity@k, vendor self-preference, and a counterfactual-delegation ceiling.

To characterize the substrate, five reference conditions are evaluated: blind (no peer description), aware-c1 (curated rubric from public sources), aware-c2 (deterministic Stage-1 statistics), aware-c3 (dual out-of-pool LLM judges: Grok-4 + Llama-4-Maverick), and aware-tool-only (C2 card available on-demand via read_profile only, no preloaded system-prompt injection). The last condition isolates delivery channel from description content.

Key Contributions

• A fixed benchmark substrate (task suite, pool, delegation interface, annotation layer, metric suite) for measuring emergent delegation, agnostic to how peer information is generated or delivered.
• A reference intervention suite: three profile-card content variants (C1 curated rubric, C2 deterministic statistics, C3 dual LLM-judge) and a delivery-channel ablation (aware-tool-only).
• Empirical characterization across 165 cells (23,375 task instances): quality flatness, delivery-channel dominance, and quantified delegation ceiling.
• Documentation of cross-vendor self-preference in delegation at 1.5–3.7× chance rate — the orchestration-tool analogue of LLM-as-judge self-preference bias.
• Full release: substrate code, annotation layer (tagger v2.0), 33 profile cards, analysis pipeline, and 220 per-condition run archives (~23k trace rows as records.jsonl.gz).

Results

• Quality flatness: Mean end-task quality is statistically indistinguishable across all four awareness conditions; mixed-effects regression yields \(|\beta| \leq 0.010\), \(p \geq 0.21\) on all aware coefficients.
• Routing fidelity@1: blind 14.2% → aware-c1 7.5% (below blind) → aware-c3 15.5% → aware-c2 20.8% → aware-tool-only 29.5%; on-demand tool access more than doubles routing precision over blind.
• Counterfactual ceiling: Perfect single-step delegation exceeds measured performance by 15–31 pp on every suite.
• Cost: aware-tool-only matches or beats blind quality at lower mean cost (avoids ~30-line per-call prompt augmentation); preloaded variants shift same-quality clusters to higher cost bands (+$0.02–0.05/task on GAIA).
• Per-agent Pareto wins: 7/11 agents on BFCL, 4/11 on GAIA, 2/11 on τ-bench have a best-aware variant Pareto-dominating blind.
• Vendor self-preference: Same-vendor delegation observed at 1.5–3.7× chance-level rate across the pool.
• Emergent taxonomy audit: 94.5% of free-form LLM re-tagging on a subsample maps cleanly to one of the 7 frozen skill labels.

Limitations

• Stage-2 solo baseline (no call_model tool at all) is not yet released; blind serves as a proxy no-delegation baseline, making pure capability-vs-delegation decomposition incomplete.
• SWE-Bench Pro excluded from headline results due to Docker harness unreliability.
• Random-slopes mixed-effects model failed to converge under four optimizers; random-intercept-only fit is conservative (inflated residual SEs).
• τ-bench delegation rate is near zero (89 invocations across 5×1,452 task-slots) due to domain-policy priming, limiting delegation signal on that suite.
• The counterfactual ceiling assumes single-step delegation at empirical Stage-1 pass rates — an optimistic bound that does not model multi-step or cascaded delegation.
• Profile cards are frozen to a single Stage-1 trace window; adaptive or continually updated peer profiles are out of scope for this substrate.

Relevance to Harnesses / Meta-Harnesses

DecisionBench is a benchmark substrate — itself a kind of meta-harness — that fixes the scaffolding (task loop, tool interface, annotation pipeline, metric computation) and then evaluates what happens when different orchestration methods (routing policies, peer-description generators) are plugged in. The call_model / read_profile interface and the deterministic annotation+metric pipeline directly mirror the concerns of meta-harness design: how to wire sub-agents together, what context to pass between them, and how to measure orchestration quality beyond end-task outcomes. The finding that delivery channel dominates description content (on-demand tool access vs. preloaded system prompt) and that routing fidelity decouples cleanly from quality are directly actionable insights for anyone building or evaluating a meta-harness that routes work across heterogeneous models.