Skip to content

EvoClaw: Evaluating AI Agents on Continuous Software Evolution

🕒 Published (v1): 2026-03-13 03:20 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvoClaw is a benchmark for evaluating LLM agents on continuous software evolution, where agents implement a dependency-constrained stream of milestones on a persistent codebase rather than isolated tasks. It is built by DeepCommit, an agentic meta-pipeline that converts noisy git histories into verifiable Milestone DAGs. Frontier models collapse from >80% on isolated tasks to at most 38.03% Score in the continuous setting.

Problem

Existing SWE benchmarks (SWE-bench, Commit0, FeatureBench) treat coding tasks as independent snapshots, resetting the environment after each task. This hides temporal dependencies and error propagation: agents can exploit shortcuts that satisfy immediate tests while silently accumulating technical debt that would block real downstream development. No benchmark measures whether an agent can sustain system integrity across a long development horizon.

Method

DeepCommit (the benchmark-construction harness) transforms raw git histories into executable Milestone DAGs through three phases:

  1. Commit history preprocessing — static analysis extracts a commit-level dependency DAG (via git blame), symbol-level modifications, and file co-change statistics; an LLM agent generates per-repo source/test directory configs to filter non-product commits.

  2. Milestone DAG construction — a four-stage LLM-agent loop:

  3. Seed Discovery: an agent selects structurally influential commits as milestone anchors using commit semantics + DAG topology (out-degree, descendant count).
  4. Milestone Consolidation: parallel sub-agents expand each seed by shared-file edits, temporal proximity, and PR/Issue references; a coordinating agent resolves overlapping claims and enforces complete commit coverage.
  5. Dependency Inference: line-level textual edges from preprocessing plus LLM-proposed symbol/co-change edges; residual edges recovered during runtime phase.

  6. Milestone Decompose: oversized milestones are split along feature boundaries using diff re-segmentation and blame remapping; targets CV < 1.0 over per-milestone LoC (achieves CV = 0.96).

  7. Runtime environment resolution — a MainAgent dispatches sub-agents (MilestoneAgent for cherry-pick + DAG repair, EnvAgent for Dockerfile generation and test collection) per milestone, producing reproducible Docker images with stable test signals; flaky tests are filtered via three repeated runs.

EvoClaw (the evaluation harness) runs agents in a continuous loop: a DAG-based scheduler unlocks milestone \(M_i\) only when all prerequisites \(\{M_j \mid M_i \text{ depends on } M_j\}\) are complete; the agent works in a persistent repository that carries forward all prior edits; upon milestone submission, the framework snapshots the repo and runs isolated evaluation asynchronously. Performance is measured by: $\(\text{Score} = \text{Recall} \times \text{Precision}\)$ where Recall = completeness of new feature implementation (F2P tests) and Precision = robustness against regressions (N2P tests), plus a strict Resolve Rate for fully completed milestones.

Key Contributions

  • Milestone-level granularity: defines a milestone as a functionally coherent, DAG-structured unit — coarser than commits (avoids noise) and finer than releases (preserves dependency structure); avg 570 LoC.
  • DeepCommit pipeline: fully automated multi-agent harness for constructing verifiable Milestone DAGs from open-source repos; 87.1% test collection success rate.
  • EvoClaw benchmark: 98 human-verified milestones across 7 Milestone DAGs, 5 programming languages, drawn from high-impact open-source repos.
  • Cross-task dependency modeling: first benchmark to expose error propagation across dependent tasks via a persistent, stateful evaluation environment.
  • Comprehensive evaluation: 12 frontier LLMs × 4 agent frameworks, revealing the continuous-vs-isolated performance gap.

Results

  • Best overall Score in continuous setting: 38.03% (Claude Opus 4.6); best in isolated: >80% for frontier models — a >42-point absolute drop.
  • Best Resolve Rate (fully completed milestones): 13.37% (Gemini 3 Pro).
  • Recall grows approximately linearly with milestone count; Precision saturates and degrades, indicating agents fail at system-level maintenance, not local implementation.
  • Unresolved regressions trigger a snowball effect: errors propagate through DAG dependency chains and eventually stall all downstream progress.
  • Agents that perform proactive codebase exploration and disciplined test verification before submission sustain higher Precision longer than blind trial-and-error agents.
  • DeepCommit achieves ≤0.10% environment-induced errors and ≤0.026% Pass-to-Fail flaky rate.
  • A single full EvoClaw evaluation costs approximately $500 with Claude Opus 4.5.

Limitations

  • Small benchmark scale: only 98 milestones across 7 repositories; generalization to a broader repository distribution is unverified.
  • DeepCommit requires human-expert guidance at key decision points during runtime environment resolution, limiting full automation.
  • Test collection success is 87.1%, not 100%; milestones with insufficient test signals are excluded, potentially biasing coverage.
  • Restricted to squash-and-merge main-branch workflows — repos using other merge strategies are not supported.
  • Cost (~$500/run) makes repeated ablations prohibitive and limits accessibility for smaller research groups.
  • Continuous evaluation with a DAG scheduler introduces coordination overhead not present in real development environments.

Relevance to Harnesses / Meta-Harnesses

DeepCommit is a direct instance of a benchmark-construction meta-harness: it orchestrates multiple specialized sub-agents (MilestoneAgent, EnvAgent, MainAgent) through a structured multi-phase pipeline with automated quality gates, runtime validation loops, and fan-out parallelism — the same architectural patterns used in general agentic meta-harnesses. The EvoClaw evaluation framework is itself a stateful agent harness that manages task scheduling via a DAG, maintains persistent execution environments, and asynchronously triggers evaluation, demonstrating how harness design choices (state persistence vs. reset, dependency-driven vs. batch dispatch) fundamentally alter what failure modes become visible. For researchers building harnesses, EvoClaw's finding that Precision saturates while Recall grows directly implicates harness design: harnesses that only reward task completion without penalizing regression will systematically overestimate agent capability. The DeepCommit pipeline also provides a reusable template for auto-generating evaluation suites from any version-controlled artifact repository.