Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

🕒 Published (v1): 2026-04-28 16:55 UTC · Source: Arxiv · link

Why this paper was selected

Observability-driven automatic evolution of coding-agent harnesses; core harness engineering

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agentic Harness Engineering (AHE) is a closed-loop framework that automatically evolves coding-agent harnesses through three structured observability pillars — component, experience, and decision — replacing manual trajectory inspection and hand-crafted edits. Ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing both human-designed harnesses and prompt-only self-evolution baselines, and the frozen harness transfers cross-benchmark and cross-model without re-evolution.

Problem

Harness engineering — designing the system prompt, tools, middleware, memory, skills, and sub-agents surrounding a coding model — is currently a manual craft that cannot keep pace with rapidly advancing base models. Automating it faces three structural obstacles: (1) a heterogeneous, untyped action space spanning multiple coupled components; (2) millions of raw trajectory tokens that bury actionable failure signal; and (3) edit effects that are hard to attribute to specific components, collapsing naïve automation into trial-and-error.

Method

AHE wraps a base model \(M\) with a fixed harness \(H\) and runs an outer loop for \(N\) iterations (Algorithm 1: rollout → clean → attribute/rollback → distill → edit → commit). Three observability pillars structure the loop:

❶ Component observability. The NexAU framework exposes seven orthogonal component types (system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, long-term memory) as files at fixed mount points in a git-versioned workspace. Each failure pattern maps to one component class; every logical edit is one git commit, giving file-level diff and rollback for free. The seed harness \(H_0\) is deliberately minimal (single shell tool, nothing else) to force attribution purity.

❷ Experience observability. The Agent Debugger framework converts raw rollout traces (∼10M tokens) into a navigable file-based environment where each trajectory message is its own file. A sub-agent explores this environment, writes a per-task root-cause report (pass/fail grounded), and aggregates all reports into a benchmark-level overview — reducing signal from millions to ∼10K tokens before the evolver reads it. Original traces are retained for drill-down verification.

❸ Decision observability. The Evolve Agent records a change manifest for every edit: failure evidence → inferred root cause → targeted fix → predicted impact (expected fixes + at-risk regressions). The following iteration intersects these predictions with observed task-level deltas, producing a per-edit verdict that either confirms or triggers rollback at file granularity. This makes every edit a falsifiable contract rather than a rationale.

A one-shot explore agent seeds a small skill library from NexAU source and public references in iteration 1; from iteration 2 onward the Evolve Agent may keep, refine, or remove these skills.

Key Contributions

• Formal definition of agent-driven harness evolution and identification of observability (not agent capability) as the bottleneck.
• Three observability pillars — decoupled file-level substrate, layered trajectory distillation, and falsifiable change manifest — that together turn harness evolution into an auditable closed loop.
• Empirical demonstration: 10 iterations lift pass@1 on Terminal-Bench 2 from 69.7% → 77.0%, surpassing Codex (71.9%), ACE (68.9%), and TF-GRPO (72.3%).
• Transfer result: frozen harness achieves highest aggregate on SWE-bench-verified (75.6%) while using 12% fewer tokens than the NexAU0 seed, and delivers +5.1–+10.1 pp cross-family gains on three alternate model families.
• Component ablation identifying tools, middleware, and long-term memory as the loci of gain, and revealing non-additive component interactions as a fundamental limit of the current loop.

Results

• Terminal-Bench 2 (89 tasks, 10 iterations): AHE 77.0% vs. NexAU0 seed 69.7% (+7.3 pp), Codex 71.9%, TF-GRPO 72.3%, ACE 68.9%; AHE is best overall, though it trails Codex on Hard (53.3% vs. 56.7%).
• SWE-bench-verified (500 tasks, cross-benchmark transfer, no re-evolution): AHE 75.6% aggregate success vs. NexAU0 75.2%, ACE 74.6%, TF-GRPO 74.2%; AHE uses 461k tokens/trial vs. ACE 679k (−32%), TF-GRPO 582k (−21%), NexAU0 526k (−12%).
• Cross-model transfer (Terminal-Bench 2, frozen harness): deepseek-v4-flash +10.1 pp (51.7%→61.8%), qwen-3.6-plus +6.3 pp (56.2%→62.5%), gemini-3.1-flash-lite-preview +5.1 pp (36.5%→41.6%), GPT-5.4 medium +2.3 pp, GPT-5.4 xhigh +2.3 pp.
• Component ablations (single-component swaps into NexAU0 seed): memory only 75.3%, tool only 73.0%, middleware only 71.9%, system prompt only 67.4% (−2.3 pp regression vs. seed); three single-component gains sum to +11.1 pp, exceeding full AHE's +7.3 pp due to non-additive interactions.
• Self-attribution (Figure 4): fix precision 33.7% vs. 11.8% random baseline (fix recall 11.8%); regression precision 5.4% and regression recall 5.6%, both near random (∼6.5% and ∼10.6% random baselines respectively).

Limitations

• Non-additive component interactions: single-component gains overshoot full AHE's aggregate; stacking memory, middleware, and system prompt causes redundant re-checks within the long-horizon turn budget, capping overall gain.
• Regression blindness in self-attribution: regression prediction precision and recall are near the random baseline (5.4%/5.6%), meaning the loop cannot reliably foresee regressions it introduces — a fundamental gap for safe autonomous evolution.
• Timeout-budget coupling: AHE's step budget and per-task timeout were fitted to GPT-5.4 high; xhigh reasoning causes more timeouts (counted as failures), discounting the gain for stronger bases.
• System prompt does not transfer: inserting the evolved system prompt alone into NexAU0 regresses −2.3 pp, indicating that prose-level strategy depends on the surrounding structural components and does not independently generalize.
• Hard-tier gap: AHE trails Codex on the 30 Hard tasks (53.3% vs. 56.7%) due to interference between AHE components on long-horizon tasks; the authors attribute this to the evolver optimizing an aggregate dominated by 55 Medium tasks.
• Single-benchmark evolution target: all results evolved on Terminal-Bench 2; interaction-aware multi-benchmark joint evolution is left as future work.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct formalization of the meta-harness concept: a harness that automatically engineers other harnesses. The three-pillar observability framework (component substrate, trajectory distillation, decision manifest) operationalizes exactly the feedback loop that manual harness engineering approximates, providing a replicable template for self-improving agent scaffolds. The result that gain concentrates in tools, middleware, and long-term memory — not the system prompt — directly challenges the dominant prompt-engineering paradigm and argues that meta-harness value lives in structural, executable components rather than natural-language instructions. The non-additive interaction finding is a critical theoretical result for anyone designing multi-component harness optimization pipelines, and the regression-prediction failure pinpoints the clearest unsolved problem (prospective regression foresight) in the harness self-evolution line of work.