CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding¶

🕒 Published (v1): 2025-11-30 21:44 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CentaurEval is a benchmark for measuring the value of human participation in AI-assisted coding by designing "collaboration-necessary" tasks that are intractable for standalone LLMs (best pass@10: 4.22%) or unaided humans (18.89% pass@1), but solvable through human-AI teaming (31.11% pass@1). It reveals that current SOTA LLMs fail at higher-order reasoning (requirement interpretation, ambiguity resolution) and that effective collaboration requires humans to act as co-reasoners, not just supervisors.

Problem¶

Existing coding benchmarks—both human-facing (LeetCode, Codeforces) and LLM-facing (HumanEval, SWE-Bench)—assume perfectly defined problems, failing to evaluate the higher-order skills (requirements engineering, problem formulation, strategic decomposition) that now define developer value in human-AI collaborative workflows. No prior benchmark quantifies the marginal contribution of human involvement in agentic coding or can distinguish collaboration gain from individual capability.

Method¶

CentaurEval constructs tasks via a 3×3 template bank (45 templates; 3 professional tracks × 3 difficulty levels) designed to be simultaneously AI-Incomplete (intractable for SOTA LLMs due to underspecified requirements, multimodal specs like UML/ER diagrams, legacy codebases, or domain-specific business logic) and Human-Reliant (manual solutions are suboptimal due to boilerplate scale, repetition, or industry-specific APIs). A GPT-4.1-powered agentic task system dynamically instantiates 450 static task instances by orchestrating four specialized tools (TechnicalParameterTool, ImplementationConstraintTool, ContextualVariableTool, InterfaceSpecificationTool). Human evaluation runs via a cloud IDE (GitHub Codespaces + VS Code + Copilot); LLM evaluation runs via an autonomous VS Code extension (CentaurEC) using Docker. Four experimental conditions are tested: CH (human-only), C0 (fully autonomous LLM), C1 (procedural intervention only), and C2 (free human-AI collaboration). A within-subject, fully counterbalanced Latin Square design with 45 expert participants compares performance across conditions.

Key Contributions¶

First benchmark explicitly designed to quantify the marginal human contribution in AI-assisted coding, framed around "collaboration-necessary" problem templates.
Dual evaluation interfaces: ecologically valid cloud IDE for humans and a reproducible 450-task automated toolkit for LLMs.
Two-stage AI-Incomplete validation protocol: template accepted only if pass@1 ≤ 5% for a commercial baseline and ≤ 10% for all SOTA models across held-out instances.
Empirical evidence that human-AI co-reasoning (not tool-use) drives gains: 51% of participants adopted fundamentally different solution approaches suggested by AI; 80% used AI for strategic brainstorming.
Auto-Calibrated Baselines for execution time and memory, enabling platform-independent binary efficiency metrics.

Results¶

LLM standalone (C0): Claude-Sonnet-4 achieves 0.67% overall pass@1, 3.11% pass@10; best model (Gemini-2.5-Pro) achieves 0.67% pass@1, 2.22% pass@10.
Procedural fix only (C1): Claude-Sonnet-4 improves to 2.89% pass@1, 4.22% pass@10—still near-zero across all models.
Human-only (CH): 18.89% pass@1.
Human-AI collaboration (C2): 31.11% pass@1 (p = 0.00739 vs. CH via paired sign test), partial pass 50.27%.
Collaboration gap is largest on hard tasks; human performance degrades with difficulty while LLM performance stays uniformly near-zero; collaborative performance remains stable across difficulty levels.
12 of 15 top-performing participants reported leveraging AI for high-level strategic reasoning.

Limitations¶

Current implementation is Python-only; language generalizability untested.
Evaluation constrained to Copilot-supported models, excluding o3, GPT-5, DeepSeek, Llama, Qwen series.
Participants are exclusively East-Asian university students/recent graduates; generalizability to industry professionals and other demographics is limited.
"Collaboration-necessity" is model-relative: tasks may become autonomously solvable as models improve, making any fixed benchmark interpretation time-sensitive.
Raw performance data converted to binary pass/fail for cross-task comparability, reducing result transparency.

Relevance to Agentic AI / LLM Agents¶

CentaurEval directly benchmarks the frontier between autonomous agent capability and human-in-the-loop value, a critical open question for anyone building or evaluating agentic coding systems. The finding that SOTA agents universally fail (<5% pass@1) on tasks requiring requirement interpretation and strategic planning quantifies a concrete ceiling for current agentic pipelines. The "co-reasoning" framing—where 51% of successful strategies originated from AI suggestions rather than human direction—challenges the standard principal-agent hierarchy assumed in most agentic system designs and has implications for how human oversight and interruption points should be structured. For researchers tracking LLM agent capability, this provides a harder and more ecologically valid evaluation surface than existing software-engineering benchmarks.