The Last Harness You'll Ever Build¶

🕒 Published (v1): 2026-04-22 18:51 UTC · Source: Arxiv · link

Why this paper was selected

Principles for building domain-specific long-horizon agentic harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This technical report proposes a two-level automated framework for harness engineering: an inner Harness Evolution Loop that iteratively optimizes a worker agent's harness \(H\) for a single task, and an outer Meta-Evolution Loop that optimizes the evolution blueprint \(\Lambda\) itself across diverse tasks. The meta-level formalism maps directly onto meta-learning, where the inner loop adapts \(H\) and the outer loop optimizes the adaptation procedure. The stated goal is zero human harness engineering for novel task domains.

Problem¶

Deploying AI agents on complex, domain-specific workflows requires painstaking, expert-driven harness engineering—designing prompts, tools, orchestration logic, and evaluation criteria from scratch for each new domain. Automated prompt-optimization methods (e.g., LLM-AutoDiff) tune individual components but do not address the full harness stack. No existing system automates harness engineering end-to-end, let alone automates the design of the harness-engineering process itself.

Method¶

Harness Evolution Loop (inner loop): Starting from an initial harness \(H^{(0)}\), the loop runs for \(K\) iterations. At each step \(k\): 1. Worker Agent \(W_{H^{(k-1)}}\) executes task \(t\) and produces execution trace \(\tau\). 2. Evaluator Agent \(V\) adversarially verifies \(\tau\) against success criteria \(S\), checks ground-truth environment state, decomposes latency (LLM vs. tool time), and outputs a structured diagnostic report plus a two-tier score (pass/fail first, execution time as tiebreaker). 3. Evolution Agent \(E\) reads the full history of \((H^{(k)}, \text{report}, \text{score}, \text{verdict})\) tuples and edits the harness—prompts, tools, orchestration logic, observation structure, model config—producing \(H^{(k+1)}\).

The best-scoring harness \(H^{(\text{best})}\) is retained across iterations; evolution always branches from \(H^{(\text{best})}\) to prevent regression accumulation.

Meta-Evolution Loop (outer loop): The entire evolution blueprint \(\Lambda = (W_H, H^{(0)}, V, E)\) is itself treated as a harness. A Meta-Evolution Agent \(E_\text{meta}\) runs the inner loop across training tasks \(T_\text{train} = \{t_1, \ldots, t_n\}\), aggregates mean scores, and modifies \(\Lambda\) to improve inner-loop convergence speed and final performance:

\[\Lambda^{(\text{best})} = \arg\max_{\Lambda}\, \mathbb{E}_{t_i \sim T_\text{train}}\!\left[\text{best score}\left(\textsc{HarnessEvolutionLoop}(t_i, \Lambda, K)\right)\right]\]

Components \(E_\text{meta}\) can modify include: evaluator/evolution agent prompts, worker observation structure, inter-agent information flows, scoring function design, and loop hyperparameters.

Key Contributions¶

Formal definition of an agent harness \(H\) as all code, configuration, and execution logic excluding model weights, with a taxonomy of harness components (prompts, tools, infrastructure, orchestration, hooks, model config).
Harness Evolution Loop (Algorithm 1): closed-loop Worker–Evaluator–Evolution agent cycle that fully automates single-task harness optimization.
Meta-Evolution Loop (Algorithm 2): outer optimization over the evolution blueprint \(\Lambda\), enabling fast harness convergence on unseen tasks without human engineering.
Formal correspondence table mapping meta-evolution to classical meta-learning (Thrun & Pratt, 1998): \(H \leftrightarrow \theta\), \(\Lambda \leftrightarrow\) adaptation procedure, inner loop \(\leftrightarrow\) gradient updates, outer loop \(\leftrightarrow\) meta-gradient update.
Evaluation protocol specifying three generalization metrics on held-out \(T_\text{test}\): convergence speed (iterations to threshold), final pass rate, and robustness (variance across tasks).

Results¶

No empirical results are reported. This is a technical report presenting the framework and algorithms; the authors explicitly state that empirical results on diverse workflows are planned as follow-up work.

Limitations¶

No empirical validation: the paper contains zero experiments, benchmarks, or ablations; all claims about performance are prospective.
Computational cost unaddressed: running \(K\) inner-loop iterations per training task, multiplied across \(|T_\text{train}|\) tasks per meta-iteration, is potentially very expensive; no analysis is provided.
Evolution Agent stability: LLM-driven code modification of an entire harness may introduce regressions or subtly broken harnesses that are difficult to detect with the two-tier scoring metric.
Evaluator prompt quality: the Evaluator Agent's adversarial verification depends on the quality of success criteria \(S\) and the evaluator prompt—both of which are themselves components of \(\Lambda\) subject to meta-optimization, creating a potential bootstrapping problem.
Scope of harness modification: it is unclear what constraints (if any) prevent the Evolution Agent from making arbitrarily large or destructive changes to \(H\) at each step.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a direct first-principles treatment of harness engineering as an optimization problem, making it highly central to anyone tracking this topic. It introduces the most explicit formalization of harness structure in the literature—decomposing \(H\) into prompts, tools, infrastructure, orchestration, hooks, and model config—and then treats the harness-engineering process \(\Lambda\) as itself a harness subject to automated optimization. The meta-evolution framing provides a principled theoretical grounding (meta-learning) for what has until now been ad hoc manual iteration. For practitioners building or extending harness frameworks (e.g., Claude Code, Codex, AdaL), this work proposes the logical endpoint: a learned \(\Lambda^{(\text{best})}\) that bootstraps effective harnesses for any new task without domain expertise.