TTHE: Test-Time Harness Evolution¶

🕒 Published (v1): 2026-07-09 05:53 UTC · Source: Arxiv · link

Why this paper was selected

Test-time harness evolution; agent behavior shaped by harness, not just model weights — novel framing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

TTHE (Test-Time Harness Evolution) adapts an LLM agent's executable harness—the control program governing tool use, context construction, and error recovery—during evaluation itself, using only unlabeled execution traces and no gold labels. A population of candidate harnesses is iteratively refined by agentic proposers and committed by a judge via execution-derived proxy signals. Across five execution-grounded benchmarks, TTHE improves fixed ReAct baselines substantially without touching model weights.

Problem¶

Deployed LLM agent harnesses are frozen after pre-deployment search on labeled development data. At test time, the harness remains static even as the evaluation stream exposes rich execution evidence—tool call patterns, runtime errors, intermediate artifacts—that a human engineer would use to revise the scaffold. No existing method adapts the persistent executable harness at test time without gold labels, weight updates, or a separate repair model.

Method¶

TTHE treats the harness \(H \in \mathcal{H}\) (a Python class) as the adaptation state. For each unlabeled test batch \(X_t\), it maintains \(G\) parallel candidate harness branches evolved over \(R\) rounds via three operators:

OBSERVE: executes every candidate harness on \(X_t\), collecting full traces (prompts, completions, tool calls, stdout/stderr, artifacts, errors) and three proxy signals: execution health \(s \in \{0,1\}\), round-trip consistency \(b \in [0,1]\) (output back-translated and scored against input without gold), and public-test pass rate \(p \in [0,1]\).
PROPOSE: \(G\) coding agents run in parallel; each proposes a code edit to its own branch while reading other branches' traces; three branches are steered toward conservative-repair, exploratory, and adversarial edit objectives for diversity.
JUDGE: an agentic selector inspects code, traces, and proxy signals (may re-execute candidates) and commits one harness \(H_{t+1}\) (Eq. 2) that persists to the next batch.

Gold labels \(y\) are strictly outside the loop, used only for post-selection measurement. The solver, proposers, and judge all invoke the same frozen backbone (deepseek-v4-flash in main runs) through different harnesses; Claude Code serves as the coding-agent scaffold for proposers and judge. Baseline is a standard ReAct harness \(H_0\).

Key Contributions¶

Formulates test-time harness evolution as population-based generate-and-judge over executable programs, with no weight updates, gold labels, or separate adaptation model.
Demonstrates improvements across five execution-grounded domains (text-to-SQL, competitive programming, software engineering, data-science coding, agentic tool-use).
Exposes harness code, execution traces, and per-batch decisions for behavioral inspection.
Provides diagnostics—search-budget, batch-size, and cross-model ablations plus per-batch trace audits—characterizing failure modes: non-monotonic gains with budget, selection regret (better candidate generated but not committed), and proxy gaming.

Results¶

BIRD (hard slice, text-to-SQL): 12.0% → 50.0% vs. fixed ReAct baseline (baseline constructed from systematic failures, so large absolute gap reflects recovery on those queries).
LiveCodeBench (hard slice, competitive programming): 30.0% → 38.3%.
SWE-bench Verified (hard slice, 40 multi-file/large-diff instances): 20.0% → 35.0% (harness evolved from mini-swe-agent baseline).
DS-1000 (hard slice, data-science code): 38.0% → 44.0%.
claw-eval (agentic tool-use): 48.9 → 69.8 mean graded score (×100 scale).
Batch-size ablation on BIRD (G=3, R=3): committed-harness accuracy is non-monotonic in \(B\), peaking at \(B=10\); fixed baseline is 12%.
Additional SQL runs with MiMo V2.5 and Kimi K2.5 corroborate cross-model generalization.

Limitations¶

Proxy signals are imperfect oracles: a harness can pass public tests while failing hidden ones, or optimize round-trip consistency while answering the wrong question—proxy gaming is a central failure mode.
Performance is non-monotonic in search budget; more candidates do not monotonically improve results.
Selection regret: the judge can commit a plausible but suboptimal program under imperfect proxy evidence even when a better candidate was generated.
Transductive scoring (harness selected and measured on the same batch) is distinct from prequential generalization; real-world deployment would require the committed harness to generalize to the next batch.
Hard slices are constructed from baseline failures and should not be used for cross-model absolute comparison.
Wall-clock and compute costs scale with \(G \times R\) executions per batch; fixed resource budgets bound but do not eliminate overfit risk.

Relevance to Agentic AI / LLM Agents¶

TTHE directly addresses a core bottleneck in deployed LLM agent systems: harnesses encode critical behavioral logic but are treated as static artifacts post-deployment. By moving harness optimization inside the evaluation loop and using execution traces as the sole learning signal, TTHE offers a label-free, weight-frozen path to continuous agent improvement—directly relevant to anyone building self-improving or self-correcting agent pipelines. The work also makes proxy reliability a first-class research object, connecting to broader challenges in reward hacking and self-evaluation in agentic RL. It sits at the intersection of automated agent design (DSPy, ADAS, AFlow), test-time adaptation, and executable self-modification (Darwin Gödel Machine, MOSS), advancing all three by operating unsupervised at deployment time.