ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents¶

🕒 Published (v1): 2026-06-30 06:06 UTC · Source: Arxiv · link

Why this paper was selected

Subagent orchestration benchmark; Cihang Xie group; directly targets agent harness patterns

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ClawArena-Team is a benchmark that isolates the subagent-management ability of a single LLM acting as a conductor: it creates, empowers, schedules, and integrates a fixed pool of specialist subagents across 41 multimodal, multi-directory scenarios. No existing benchmark combines runtime subagent creation, modality routing, least-privilege enforcement, asynchronous scheduling, and dynamic workflow authoring under execution-based scoring. Across twelve models, the key finding is that privilege granting—not perception—is the primary management bottleneck, and API cost is decoupled from management quality.

Problem¶

Existing benchmarks either score a single agent's own reasoning (SWE-bench, GAIA, τ-bench) or evaluate a multi-agent system with fixed, predefined roles—neither isolates a single LM's ability to create subagents at runtime, grant them least-privilege access, route tasks by modality, schedule them asynchronously, and author dynamic workflows. The management skill of the LLM-as-manager is therefore largely unmeasured.

Method¶

The benchmark frames subagent management as a principal–agent problem under information and capability asymmetry. The main agent (principal) perceives only text and accesses only part of the workspace; it must accomplish tasks by exercising six operations: creation, modality routing, least-privilege empowerment, scheduling, dynamic workflow authoring, and integration. The subagent pool is fixed and locally served via vLLM (LLM/VLM backed by gemma-4-31b-it, OMNI by gemma-4-e4b-it), so every manager commands identical workers and score differences reflect management quality alone.

Benchmark construction uses workflow-driven parallel authoring: a controller fans out to isolated subagent-authors and accepts a scenario only after a real baseline management run classifies every failure as true-difficulty rather than brittle evaluation artifact. Each scenario satisfies ten hard design constraints (C1–C10) targeting distinct failure modes (e.g., C5 forces non-text modality delegation, C6 forces parallel scheduling, C9 plants text decoys next to real multimodal assets).

Scoring is fully execution-based (shell exit codes, no LLM judge). Five per-run components are reported: - $\text{TCR}$: task completion rate - $\text{TPP}$: tool-permission precision (granted tools actually used) - $\text{ROC}$: read-only compliance - $\text{WPP}$: workspace-permission precision (accessed files / granted files) - $\text{MCA}$: modality-choice accuracy

The composite Subagent-Management Score is:

\[\text{SMS} = \text{TCR} \times \frac{\text{TPP} + \text{ROC} + \text{WPP} + \text{MCA}}{4}\]

Since the management factor $\in [0,1]$, $\text{SMS} \leq \text{TCR}$ always: sloppy management penalizes task score, but correct management cannot inflate it.

Key Contributions¶

First benchmark to jointly score a single LM on runtime subagent creation, modality routing, least-privilege empowerment, asynchronous/background scheduling, dynamic workflow authoring, and result integration under execution-based scoring.
41 multimodal, multi-directory scenarios (258 evaluation rounds, 72 staged updates, 170.5 MiB workspaces with 28.9 M tokens) spanning law, medicine, engineering, business, and science.
The SubagentManagement Score (SMS) composite metric that discounts task correctness by privilege and routing precision.
Empirical evaluation of 12 proprietary and open-weight models revealing three findings: (1) privilege granting is the bottleneck, (2) cost and management quality are decoupled, (3) leaderboard scores cluster while orchestration behaviors diverge by over an order of magnitude.
A self-referential construction pipeline: the benchmark is itself authored via the subagent-management paradigm it measures.

Results¶

Top performer: claude-fable-5 leads with SMS = 60.0% (TCR = 74.4%), at ≈$93/run.
Second: gemini-3.5-flash at SMS = 53.8%, ≈$23.7/run.
Best open-weight: glm-5.2 at SMS = 50.4%, $22.9/run (fourth overall).
10 of 12 models cluster within a 9.9-point SMS band (43.9–53.8%); only claude-fable-5 exceeds SMS = 54%.
No model exceeds 50% WPP (workspace-permission precision); subagents are routinely granted roughly 2× the files they actually touch.
ROC and MCA are nearly saturated (≥92% for all 11 capable models), confirming that the bottleneck is privilege precision, not modality routing.
Cost spans >100× ($0.8–$93/run) while SMS spans <4× (15.3–60.0%); deepseek-v4-pro achieves SMS = 46.4% at $1.7 via cache-read discounts.
Per-subagent forbidden-access rate (SAFa) diverges ~12× among capable models (0.48 for gpt-5.5 to 5.78 for deepseek-v4-pro) despite similar SMS values.
Dynamic-workflow invocations range from 8 to 112 across models; background scheduling and session continuation are unevenly adopted.
No scenario is solved by all models; full-pass rate (all rounds correct in a scenario) is near zero across the board.

Limitations¶

Subagent pool is fixed (gemma-4-31b-it / gemma-4-e4b-it) and locally served; findings on management skill are conditioned on this specific worker capability.
41 scenarios and 258 rounds, while carefully constructed, may not generalize to all domains or workspace types.
A single evaluation run per model is reported; stochastic variance across runs is not quantified.
API costs for self-hosted models are proxied by OpenRouter reference rates, not actual compute cost, complicating fair cost comparisons.
The benchmark cannot distinguish whether management failures (e.g., over-granting) arise from inability versus deliberate strategy by the model.
Text truncated before full error analysis; stale-belief failures across staged updates and integration failures are described only partially.

Relevance to Agentic AI / LLM Agents¶

ClawArena-Team directly addresses a critical gap in agentic AI evaluation: moving beyond single-agent benchmarks to measure the management layer that production multi-agent systems require, where a single LLM must orchestrate a heterogeneous subagent team dynamically rather than following predefined roles. The finding that privilege granting is the primary bottleneck (WPP never reaching 50%) is directly actionable for agent safety research—over-broad path and tool grants are the dominant failure mode, not perception or routing errors. The cost–quality decoupling result (>100× cost range with <4× SMS range) has practical implications for agent system design, suggesting that cheaper open-weight models can reach Pareto-efficient management without flagship-level spend. The benchmark's execution-based, no-LLM-judge scoring methodology sets a reproducible standard for future agentic evaluation.