Skip to content

Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents

🕒 Published (v1): 2025-02-27 13:31 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Collab-Overcooked is a multi-agent benchmark built on Overcooked-AI that enforces strict collaboration between LLM agents via resource isolation and asymmetric task knowledge. It introduces process-oriented metrics (TES, ITES) beyond end-to-end success rates to diagnose where collaboration breaks down. Experiments on 13 LLMs reveal that initiating collaboration—not responding to it—is the primary bottleneck.

Problem

Existing LLM-MAS benchmarks (CuisineWorld, VillagerBench, RocoBench) conflate collaboration capability with task-completion rate, allow agents to succeed without collaborating, and rely exclusively on end-to-end metrics that obscure whether collaboration itself drove performance. No prior benchmark enforces collaboration as a hard constraint or provides fine-grained, process-oriented diagnostics.

Method

The benchmark extends Overcooked-AI to a two-agent chef-and-assistant setup with two structural enforcement mechanisms: resource isolation (each agent has access to disjoint resources, requiring counter-mediated exchange) and asymmetric task knowledge (only one agent knows the recipe, so communication is mandatory). Thirty sequential tasks across 6 complexity levels (stratified by minimum required collaborative actions) are evaluated using three metrics:

  • Progress Completeness (PC): \(\text{PC} = \frac{1}{K}\sum_{k=1}^K \text{TES}(h_k)\), a TES-based aggregate over all agents penalizing redundancy.
  • Trajectory Efficiency Score (TES): \(\text{TES}(h_k) = \max_j \frac{(1+\beta^2)D^j_{\max}(h_k, g^j_k)}{m_k + \beta^2 n_k}\), comparing agent history against referential action trajectories (RATs) via longest order-preserving subsequence, with redundancy penalty \(\beta\).
  • Incremental TES (ITES): \(\text{ITES}(a, h_k) = \text{TES}(h_k \cup a) - \text{TES}(h_k)\), measuring the marginal task-progress contribution of each individual action; used to define binary Initiating Capability (IC) and Responding Capability (RC).

A common in-context-learning baseline with memory and reflection is used uniformly across all 13 models.

Key Contributions

  • Open-source benchmark with 30 tasks Ă— 6 complexity levels enforcing collaboration as a hard structural requirement (not merely incentivized).
  • Formal decomposition of collaboration capability into initiating and responding, each measured by IC/RC derived from ITES.
  • Process-oriented metrics (TES, ITES, PC) that diagnose trajectory-level inefficiency rather than binary success.
  • Comprehensive evaluation of 13 LLMs (7B–671B+, open and closed source) revealing attention misalignment as a key failure mode for collaboration initiation.

Results

  • Claude Sonnet 4 achieves the highest overall SR: 100/100/96/92/74/58% across levels 1–6; strongest on high-complexity tasks.
  • DeepSeek-R1 is best open-source model (100/100/98/82/44/30% SR), but consumes 18.6Ă— the tokens of GPT-4o.
  • GPT-3.5-turbo and models ≤8B parameters collapse at level 2 (SR: 8% and 0–4%).
  • All models show RC > IC, confirming instruction-following supports responding but initiation is the bottleneck.
  • Human participants maintain near-ceiling SR and PC across all 6 levels even under time constraints; DeepSeek-R1 degrades sharply above level 3.
  • Performance diverges sharply above level 4 for all models, including top-tier ones.

Limitations

  • Two-agent setup only; scaling to \(N>2\) agents is not evaluated.
  • Benchmark domain is restricted to kitchen/cooking tasks; transfer to other domains is unproven.
  • RATs are manually annotated for 30 tasks; coverage is finite and may not capture all valid solution paths.
  • Asymmetric task knowledge and resource isolation are the only forced-collaboration mechanisms; other real-world collaboration dependencies are not modeled.
  • Token budget analysis is limited to a few models; cost-efficiency trade-offs are not systematically studied.

Relevance to Harnesses / Meta-Harnesses

Collab-Overcooked is an evaluation harness for multi-agent LLM systems: it provides a structured environment, a fixed task suite, standardized agent architecture, and a metrics pipeline (TES/ITES/IC/RC) that can be reused across any LLM backend—precisely the scaffolding pattern that meta-harnesses build on. The paper's uniform baseline agent (memory + reflection + error correction) is itself a lightweight agent harness applied uniformly to 13 models, demonstrating how a shared harness enables apples-to-apples comparisons. The open-source release of environments, tasks, and the evaluation package makes it composable with broader harness pipelines that orchestrate model evaluation at scale.