Skip to content

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

🕒 Published (v1): 2026-07-23 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ICAE-Bench is a 480-task benchmark for evaluating coding agents under interactive, "vibe-coding" settings where agents must resolve fuzzy product requirements through multi-turn clarification with an automated User Agent before building a complete repository from scratch. Unlike prior benchmarks that supply fully specified instructions, ICAE-Bench derives controlled ambiguity from real open-source repositories and evaluates generated repos via language-agnostic black-box tests. Experiments across six coding models and two agent frameworks reveal that agents can reproduce visible behavior but consistently fail on hidden constraints, boundary cases, and long-horizon integration.

Problem

Existing coding benchmarks assume static, fully specified tasks (function completion, patch generation, or from-scratch generation with explicit PRDs), but emerging "vibe-coding" workflows require agents to handle incomplete product intent, elicit missing requirements through dialogue, and integrate clarified information across a full repository build. No prior benchmark evaluates this interactive project-building capability under controlled, reproducible conditions.

Method

Each benchmark instance \(T = (D_f, E, P, U, B)\) is constructed from a real open-source repository whose original tests pass in Docker:

  1. GroundPRD synthesis: An LLM constructs a complete product requirement document and refactors original tests into standardized black-box JSON cases (Native suite); an Enhanced suite adds boundary/corner-case coverage synthesized and validated against the golden implementation.
  2. Fuzzification: GroundPRD is progressively degraded into three fuzzy levels — L1 (minimal info, PM-style stub), L2 (partial restoration), L3 (light redaction) — by hiding API commitments, edge-case constraints, and architectural requirements into structured User Agent Data.
  3. Ultimate image: A Docker image is provisioned with the golden repo's dependency stack but with golden code, original tests, and hidden artifacts removed, preventing leakage during evaluation.
  4. Grounded User Agent: Answers clarification questions strictly from benchmark-authored requirement records (no free-form hallucination); a fixed question budget controls interaction.
  5. Multi-dimensional evaluation: Black-box pass rate (Public, Native, Enhanced cases) plus structural metrics (file count, LOC, method/class/API similarity), agentic metrics (semantic similarity, design quality), and interaction metrics (constraint coverage, fallback rate, budget usage).

Artifact validity is confirmed by iterative reconstruction from GroundPRD using Claude-Opus-4.8, plus GPT-5.5 semantic-equivalence checks (mean similarity scores 0.952 for L1, 0.942 for L3 on ICAE-Bench-Lite).

Key Contributions

  • Formulation of interactive project generation from fuzzy product intent as a benchmark setting, distinct from static 0-to-1 generation.
  • A construction pipeline that derives controlled ambiguity from verifiable open-source repositories, preserving evaluability without sacrificing realism.
  • ICAE-Bench: 480 tasks across 12 languages (C#, C++, Dart, Go, Java, JavaScript, TypeScript, Kotlin, PHP, Python, Ruby, Rust) with language-agnostic black-box evaluation; ICAE-Bench-Lite (50 tasks) for fast ablations.
  • A grounded User Agent design that prevents requirement invention and source leakage during multi-turn clarification.
  • A multi-dimensional evaluation protocol covering functional correctness, semantic/API similarity, structural fidelity, design quality, and interaction quality.

Results

(Paper text truncated before full quantitative results section; the following are stated in the text.)

  • GroundPRD performance serves as a strong upper bound; interaction with the User Agent recovers only part of the gap between fuzzy and fully specified settings.
  • Higher constraint coverage (more questions asked/answered) does not automatically translate into higher pass rates — the bottleneck is preserving clarified requirements and coherently integrating them into a repository implementation.
  • Agents typically reproduce visible/public behavior but fail on hidden constraints, boundary cases, and long-horizon integration across the generated codebase.
  • Evaluated on six coding models within Claude Code and OpenHands frameworks.

Limitations

  • Results section is not fully available in the provided text; specific per-model and per-language pass-rate numbers cannot be reported.
  • Ambiguity levels (L1–L3) are operationally defined per-task and described explicitly as not a universal psychometric scale, limiting generalizability of level-to-level comparisons across tasks.
  • Construction relies on LLM-assisted GroundPRD synthesis and fuzzification, introducing potential inconsistencies despite the verification pipeline.
  • The User Agent's fixed question budget and keyword-triggered retrieval may not fully capture the range of real human–agent clarification dynamics.
  • Enhanced cases are synthesized rather than sourced from real usage, potentially missing naturalistic failure modes.
  • Evaluation of very large repositories (max golden LOC ~2.9M) may be dominated by scaffolding rather than implementation intelligence.

Relevance to Agentic AI / LLM Agents

ICAE-Bench directly benchmarks the multi-capability loop — planning, clarification, tool use, debugging, iterative repository construction — that characterizes frontier coding agents operating in vibe-coding settings, going beyond the patch-or-complete framing of SWE-bench and its variants. The grounded User Agent design is a methodological contribution to controlled interactive evaluation, relevant to any benchmark that needs reproducible human-in-the-loop simulation. The finding that constraint retrieval does not bottleneck agents but coherent long-horizon integration does is an actionable signal for agent architecture research (memory, context management across tool calls). The 12-language, language-agnostic black-box evaluation also enables cross-lingual agent capability assessment, which is underexplored in the agentic coding literature.