Rethinking the Evaluation of Harness Evolution for Agents¶

🕒 Published (v1): 2026-07-14 00:18 UTC · Source: Arxiv · link

Why this paper was selected

Exposes train/eval leakage in harness evolution benchmarks; reshapes how agent harness progress is measured

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper identifies two fundamental flaws in how automatic harness evolution for LLM agents is evaluated: the search procedure itself is a form of test-time scaling and must be compared against matched-budget baselines, and using the same benchmark for harness search and final evaluation risks overfitting. Experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 show that harness evolution does not consistently outperform simple test-time scaling and generalizes poorly to held-out tasks.

Problem¶

Existing automatic harness evolution methods (which iteratively improve the prompts, tools, memory, and control logic surrounding an LLM agent) report gains on the same public benchmarks used to guide the search. This creates two conflated confounds: (1) the iterative-search budget in harness evolution is directly analogous to test-time scaling compute, so any gain could arise purely from additional inference rather than better harness design; (2) evaluating on search-overlapping tasks means reported improvements may encode task-specific shortcuts rather than generalizable scaffold improvements.

Method¶

The authors formalize a unified budget protocol parameterized by compute budget \(K\), feedback type (no unit tests vs. unit tests \(g\)), and what the budget updates (trajectories vs. harness). Four methods are compared under the same \(K=5\), same initial harness \(h_1\), and same rollout budget \(m=1\):

Parallel Sampling: draw \(K\) i.i.d. trajectories \(y_1,\ldots,y_K \sim \pi_\theta(\cdot|x;h)\); select via self-judge or unit-test oracle.
Sequential Refinement: chain trajectories so each \(y_k \sim \pi_\theta(\cdot|x,\Phi(y_{k-1});h)\); depth-first search over solution space.
Harness Evolution (instantiated as AHE with explore agent disabled): a meta-agent \(M\) reads summarized experience \(\Phi(C_{k-1})\) over a task batch and emits \(h_k = M(\Phi(C_{k-1}))\); optimizes a shared harness across tasks.
Harness Scaling (new baseline): instance-guided adaptation where \(h_k = M(x, \Phi(h_{k-1}, y_{k-1}))\) per task, producing a task-specific evolving harness rather than a reusable one.

Experiments run on Terminal-Bench 2.1 (89 terminal tasks) across Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini, averaged over two independent runs per condition.

Key Contributions¶

Identifies and formalizes two evaluation flaws in the harness evolution literature: uncontrolled search-budget comparison and benchmark overlap between search and evaluation.
Introduces harness scaling as a conceptually important new baseline that isolates instance-level harness adaptation from dataset-level evolution.
Provides a unified budget framework specifying what each method updates and what feedback it consumes.
Demonstrates empirically that harness evolution does not consistently outperform test-time scaling under matched budgets.
Shows that evolved harnesses exhibit severe overfitting: marginal gains (+0.6 pp average) on held-out tasks vs. 69.9→73.0 apparent gains when evaluated on search tasks.

Results¶

Without unit tests (pass@1): - Direct sampling baseline: 68.2 avg - Parallel Sampling: 72.3 avg (best) - Sequential Refinement: 69.3 avg - Harness Evolution: 67.4 avg (below baseline) - Harness Scaling: 71.8 avg

With unit tests (pass@1 / pass@5): - Parallel Sampling: 86.0 / 86.0 avg - Sequential Refinement: 84.3 / 91.8 avg (best pass@5) - Harness Evolution: 75.8 / 86.2 avg - Harness Scaling: 82.6 / 89.3 avg

Generalization (held-out test set, unit tests used for training only): - Initial harness: 67.7 avg - Harness Evolution: 68.3 avg (+0.6 pp); +1.2 on Claude Opus 4.6, +0.0 on GPT-5.4

Limitations¶

Evaluation confined to Terminal-Bench 2.1, which the authors acknowledge may not be harness-sensitive (a minimal shell tool + basic prompt already suffices for most solvable tasks).
Budget \(K=5\) is small; harness evolution may require larger budgets to manifest benefits if they exist.
Only one harness evolution method (AHE) is instantiated; Meta-Harness and AEVO are cited but not directly compared under the controlled protocol.
High baseline performance on Terminal-Bench leaves little headroom; remaining failures may be model-bottlenecked, not harness-bottlenecked.
The paper does not test on benchmarks explicitly designed for harness sensitivity (e.g., tasks requiring specialized tools or domain-specific workflows).

Relevance to Agentic AI / LLM Agents¶

The paper directly challenges a growing line of work that treats automatic scaffold/harness optimization as a path to better agents, arguing that the compute spent on harness search is better attributed to test-time inference rather than scaffold improvement. For researchers building or evaluating agentic systems, this raises a concrete methodological requirement: any harness evolution claim must beat matched-budget parallel sampling and sequential refinement, and must be validated on held-out tasks. The introduction of harness scaling as a baseline is practically important, as it separates reusable harness design from instance-level adaptation—a distinction that will sharpen future ablation designs in agentic scaffolding research.