TTHE: Test-Time Harness Evolution¶

🕒 Published (v1): 2026-07-09 05:53 UTC · Source: Arxiv · link

Why this paper was selected

Core topic: defines harness as executable scaffold; proposes test-time harness evolution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

TTHE (Test-Time Harness Evolution) adapts an LLM agent's executable harness—the control program governing tool use, context construction, and error recovery—during evaluation itself, using only unlabeled execution traces and no gold labels. A population of candidate harnesses is iteratively refined by agentic proposers and committed by a judge via execution-derived proxy signals. Across five execution-grounded benchmarks, TTHE improves fixed ReAct baselines substantially without touching model weights.

Problem¶

Deployed LLM agent harnesses are frozen after pre-deployment search on labeled development data. At test time, the harness remains static even as the evaluation stream exposes rich execution evidence—tool call patterns, runtime errors, intermediate artifacts—that a human engineer would use to revise the scaffold. No existing method adapts the persistent executable harness at test time without gold labels, weight updates, or a separate repair model.

Method¶

TTHE treats the harness \(H \in \mathcal{H}\) (a Python class) as the adaptation state. For each unlabeled test batch \(X_t\), it maintains \(G\) parallel candidate harness branches evolved over \(R\) rounds via three operators:

OBSERVE: executes every candidate harness on \(X_t\), collecting full traces (prompts, completions, tool calls, stdout/stderr, artifacts, errors) and three proxy signals: execution health \(s \in \{0,1\}\), round-trip consistency \(b \in [0,1]\) (output back-translated and scored against input without gold), and public-test pass rate \(p \in [0,1]\).
PROPOSE: \(G\) coding agents run in parallel; each proposes a code edit to its own branch while reading other branches' traces; three branches are steered toward conservative-repair, exploratory, and adversarial edit objectives for diversity.
JUDGE: an agentic selector inspects code, traces, and proxy signals (may re-execute candidates) and commits one harness \(H_{t+1}\) (Eq. 2) that persists to the next batch.

Gold labels \(y\) are strictly outside the loop, used only for post-selection measurement. The solver, proposers, and judge all invoke the same frozen backbone (deepseek-v4-flash in main runs) through different harnesses; Claude Code serves as the coding-agent scaffold for proposers and judge. Baseline is a standard ReAct harness \(H_0\).

Key Contributions¶

Formulates test-time harness evolution as population-based generate-and-judge over executable programs, with no weight updates, gold labels, or separate adaptation model.
Demonstrates improvements across five execution-grounded domains (text-to-SQL, competitive programming, software engineering, data-science coding, agentic tool-use).
Exposes harness code, execution traces, and per-batch decisions for behavioral inspection.
Provides diagnostics—search-budget, batch-size, and cross-model ablations plus per-batch trace audits—characterizing failure modes: non-monotonic gains with budget, selection regret (better candidate generated but not committed), and proxy gaming.

Results¶

BIRD (hard slice, text-to-SQL): 12.0% → 50.0% vs. fixed ReAct baseline (baseline constructed from systematic failures, so large absolute gap reflects recovery on those queries).
LiveCodeBench (hard slice, competitive programming): 30.0% → 38.3%.
SWE-bench Verified (hard slice, 40 multi-file/large-diff instances): 20.0% → 35.0% (harness evolved from mini-swe-agent baseline).
DS-1000 (hard slice, data-science code): 38.0% → 44.0%.
claw-eval (agentic tool-use): 48.9 → 69.8 mean graded score (×100 scale).
Batch-size ablation on BIRD (G=3, R=3): committed-harness accuracy is non-monotonic in \(B\), peaking at \(B=10\); fixed baseline is 12%.
Additional SQL runs with MiMo V2.5 and Kimi K2.5 corroborate cross-model generalization.

Limitations¶

Proxy signals are imperfect oracles: a harness can pass public tests while failing hidden ones, or optimize round-trip consistency while answering the wrong question—proxy gaming is a central failure mode.
Performance is non-monotonic in search budget; more candidates do not monotonically improve results.
Selection regret: the judge can commit a plausible but suboptimal program under imperfect proxy evidence even when a better candidate was generated.
Transductive scoring (harness selected and measured on the same batch) is distinct from prequential generalization; real-world deployment would require the committed harness to generalize to the next batch.
Hard slices are constructed from baseline failures and should not be used for cross-model absolute comparison.
Wall-clock and compute costs scale with \(G \times R\) executions per batch; fixed resource budgets bound but do not eliminate overfit risk.

Relevance to Harnesses / Meta-Harnesses¶

TTHE directly addresses the central challenge in meta-harness design: how to adapt the control layer (tool routing, context construction, error recovery) without retraining the underlying model or requiring labeled outcomes. Its use of execution-derived proxy signals — traces alone, no gold labels — makes it deployable against any black-box LLM, which is exactly the constraint practitioners face when building harnesses over hosted models. The population-based proposer/judge loop is essentially an automated harness search algorithm, making it a concrete blueprint for self-improving scaffolds rather than a one-shot design. For someone building or evaluating agent harnesses, TTHE is a direct methodological contribution: it shows that harness quality is itself optimizable at test time, shifting harness design from a static engineering decision to a dynamic, evidence-driven process.