Skip to content

TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

🕒 Published (v1): 2026-07-02 17:35 UTC · Source: Arxiv · link

Why this paper was selected

Executable live benchmark for test-code co-evolution; targets coding-agent eval gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TestEvo-Bench is an executable, live benchmark for evaluating LLM/agent systems on test-and-code co-evolution tasks mined from real commit histories. It provides two tracks—test generation and test update—with execution-grounded metrics (success rate, coverage, mutation score) and a timestamp-anchored design that supports contamination-aware evaluation. State-of-the-art agents (Claude Code, Gemini CLI, SWE-Agent) reach up to 77.5% success on test generation and 74.6% on test update, but performance degrades on recent tasks and under cost constraints.

Problem

Existing test generation and update benchmarks either target a fixed code snapshot (not a code change), rely on static diff signals without execution validation, or cover only one of the two co-evolution patterns (new tests vs. updated tests). This prevents rigorous evaluation of whether an agent truly understands how a code change should propagate into the test suite—i.e., whether the generated/updated test is semantically tied to the behavioral delta, not merely syntactically plausible.

Method

A three-phase automated pipeline mines Java Maven repositories via GitHub Search API: 1. Mining: Walk adjacent commit pairs that compile and pass tests on both endpoints; extract method-level diffs and per-test runtime dependencies via lightweight dynamic instrumentation. 2. Execution-backed cleaning: Rebuild each candidate revision pair and execute tests across four cross-revision triples (\(T_{old} \rightarrow C_{old}\), \(T_{new} \rightarrow C_{new}\), \(T_{old} \rightarrow C_{new}\), \(T_{new} \rightarrow C_{old}\)); filter out refactorings (via RefactoringMiner), concurrent test-directory changes, and revision-pair gaps >12 hours. 3. Task construction: Label tasks as test generation (new test must pass on \(C_{new}\), fail on \(C_{old}\)) or test update (old test fails on \(C_{new}\); agent must produce a passing replacement). Tasks are grouped by commit pair; each ships old/new revisions, focal code methods, and execution metadata.

Evaluation applies agent-produced patches, recompiles the affected Maven module, runs tests in an isolated JVM, and collects JaCoCo coverage and Universal Mutator mutation scores restricted to focal dependency lines. The primary metric, Success%, requires a test to pass on the new revision and (for generation) fail on the old.

Key Contributions

  • First live and largest executable benchmark for test-and-code co-evolution: 746 generation + 509 update tasks from 152 open-source Java projects (screened from 59,950 candidates).
  • Execution-backed task labels—ground truth is cross-revision test outcomes, not static diff heuristics.
  • Reproducible evaluation framework with compilation, pass/fail, line coverage (JaCoCo), and mutation score (Universal Mutator) metrics.
  • Timestamp-anchored, contamination-aware evaluation: tasks are filterable by the model's training cutoff using half-year subsets.
  • Baseline experiments on four harness–model configurations: Claude Code + Opus 4.7, Gemini CLI + Gemini 3.1 Pro, SWE-Agent + Opus 4.7, SWE-Agent + Gemini 3.1 Pro.

Results

Test generation (746 tasks, 1,961 methods): - Claude Code + Opus 4.7 and Gemini CLI + Gemini 3.1 Pro: 77.5% Success% (best) - SWE-Agent + Gemini 3.1 Pro: 68.6%; SWE-Agent + Opus 4.7: 66.1% - Redundant% (passes new but not fail-on-old): 17.4%–19.9% across all agents—agents generate passing tests but often fail to produce semantically discriminating oracles - CovOnPass: 74.3%–78.0%; MutOnPass: 55.0%–57.1% - SWE-Agent HrnsFail% (harness incompatibility): 10.1%–14.1% vs. 0% for industrial agents

Test update (509 tasks, 1,138 methods): - Gemini CLI: 74.6%, Claude Code: 74.4%, SWE-Agent + Gemini 3.1 Pro: 73.9%, SWE-Agent + Opus 4.7: 65.6% - CovOnPass: 79.1%–79.4%; MutOnPass: 44.6%–46.0% - SWE-Agent + Opus 4.7 HrnsFail%: 11.0% vs. ≤3.5% for others

Temporal trend: Success% is materially lower on the most recent half-year subsets (post-2024) and drops significantly when per-task cost is capped—indicating both data-leakage effects and cost sensitivity.

Limitations

  • Restricted to Java/Maven repositories; generalizability to other languages/build systems is untested.
  • Mutation analysis (Universal Mutator) is restricted to focal dependency lines; broader code quality signals are not measured.
  • The 12-hour revision-pair gap filter and conservative cleaning pipeline discard a large fraction of candidates (59,950 → 13,868 classified → 1,255 final tasks), which may bias toward simpler or cleaner commits.
  • Agents are evaluated at default (uncapped) cost; cost-limited results show significant degradation but are not fully characterized.
  • The benchmark is currently Java-only and evaluates unit-test level changes; integration tests and multi-module co-evolution patterns are out of scope.
  • No human validation pass analogous to SWE-bench Verified is described for ambiguous or borderline tasks.

Relevance to Agentic AI / LLM Agents

TestEvo-Bench directly benchmarks the full software-engineering agent loop—repository navigation, code understanding, test synthesis, iterative compilation/execution feedback—making it a natural complement to SWE-bench for evaluating coding agents beyond bug fixing. The live, timestamp-anchored design addresses a critical weakness of static agent benchmarks: contamination by training data, which inflates apparent capability. The finding that industrial co-designed harness–model systems (Claude Code, Gemini CLI) outperform the modular SWE-Agent by ~10 pp on generation underscores how harness–model integration affects agentic task completion at the tool-use layer, not just the model layer. The cost-sensitivity result is directly actionable for agent infrastructure design: success rate degrades sharply under budget constraints, motivating research into more token-efficient agentic scaffolds for software maintenance tasks.