Skip to content

Natural-Language Agent Harnesses

🕒 Published (v1): 2026-03-26 17:58 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Natural-Language Agent Harnesses (NLAHs)—editable policy documents that encode agent harness logic in natural language—and Intelligent Harness Runtime (IHR), a shared runtime that interprets NLAHs into auditable agent calls, state updates, validation gates, and artifact contracts. Across coding (SWE-bench Verified), terminal-use (Terminal-Bench 2.0), and computer-use (OSWorld) benchmarks, IHR-executed NLAHs achieve task performance comparable to native code harnesses while compressing harness policy by 20–75× in token count and enabling clean module-level ablations.

Problem

Agent harnesses—the external execution systems that govern what a model sees, what tools it may call, how state is managed, when validation runs, and how failures are recovered—are almost always implemented as tightly coupled controller code. This conflates harness policy (the reusable design decisions) with harness mechanism (tool adapters, parsers, sandboxing), making harnesses opaque, non-transferable, and resistant to ablation or comparison as research objects.

Method

The system has four layers: 1. Base agent: a minimal LLM loop with only a terminal tool; child agents are launched via terminal subprocesses. 2. Runtime policy (IHR): a fixed natural-language charter that tells the base agent how to interpret and execute an NLAH—it is intentionally thin and not benchmark-specific. 3. NLAH: a per-harness natural-language document specifying stages, role contracts, state rules, evidence requirements, verification rules, recovery conditions, and stopping criteria. 4. Scripts/adapters: deterministic code for exact operations (test runners, parsers, sandboxing, benchmark adapters).

The key separation: natural language carries policy; code and the runtime carry mechanism. Even a nominally single-agent harness is realized as an IHR orchestrator plus one executor child, keeping the harness boundary visible. NLAH writing principles emphasize stating task contracts first, separating stages from mechanisms, making state and evidence explicit, and writing module boundaries so they can be cleanly ablated.

The paper compares three realizations of each harness: (1) native Code harness, (2) Prompted NLAH (same text as passthrough instruction to Codex CLI without IHR runtime semantics), and (3) IHR-executed NLAH. Runs use Codex CLI 0.123.0, GPT-5.4-mini at reasoning effort xhigh, in Docker on Ubuntu 24.04.

Key Contributions

  • Introduces NLAHs as a new class of representation object for agent harness patterns, distinct from prompts and from controller code.
  • Introduces IHR, a shared runtime that gives natural-language harness policy a common execution substrate across benchmark families.
  • Provides a natural-language/code boundary analysis mapping eleven harness engineering aspects (agent loops, context engineering, validation, retry, budget control, etc.) to their NLAH vs. code carriers.
  • Provides controlled empirical evidence across three benchmark families for harness realization (RQ1), mechanism realization (RQ2), and module ablation (RQ3).
  • Defines new harness-engineering metrics (Artifact Contract, Tool Call Success, Failed Tool Continuation, Orchestration Reliability, Information Handoff Recall, Verification Signals, Stage Coverage, Ordered Workflow).

Results

RQ1 — Task performance (Table 1): - Live-SWE (SWE-bench Verified): Code=67.0, Prompted=77.0, NLAH=73.0 - MHTBA (Terminal-Bench 2.0): Code=36.0, Prompted=57.3, NLAH=53.9 - SeeAct (OSWorld): Code=47.1, Prompted=47.9, NLAH=46.3

RQ1 — Policy conciseness (Table 2): - Live-SWE: 60.1k tokens / 68 files → 2.9k tokens / 3 files - MHTBA: 10.5k / 3 files → 0.8k / 1 file - SeeAct: 47.5k / 5 files → 1.4k / 1 file

RQ2 — Mechanism realization (Table 4, IHR-NLAH): - Live-SWE: Artifact Contract=1.000, Tool Call Success=0.933, Failed Tool Continuation=0.992 - MHTBA: Artifact Contract=0.955, Tool Call Success=0.928, Failed Tool Continuation=0.995 - Information Handoff Recall drops to 0.322 (Live-SWE) and 0.553 (MHTBA) under parent-child execution vs. 1.0 for Prompted

RQ3 — Module ablation (Table 5, SWE Verified / OSWorld vs. Basic): - File-backed state: +2.6 / +13.9 - Self-evolution: +5.8 / +8.4 - Evidence-backed answering: +2.8 / +2.8 - Multi-candidate search: −1.6 / +2.8 (agent calls: 1.1→5.7 on SWE) - Context compression: −1.0 / −8.3

Limitations

  • IHR adds significant overhead in LLM calls, tool calls, and tokens vs. code harnesses (e.g., Live-SWE: 23.3→41.0 LLM calls, 17.7→63.4 tool calls).
  • Information handoff across parent-child boundaries is a systematic weakness: Handoff Recall of 0.32–0.55 vs. 1.00 for in-context prompting.
  • Orchestration Reliability under IHR (0.83–0.85) is lower than Prompted variants (0.99–1.00).
  • Multi-candidate search module is too expensive and infrastructure-sensitive to reliably improve outcomes under current runtime budget.
  • Context compression and Markdown memory can hurt performance when their internal success signal drifts from the benchmark's acceptance condition.
  • Experiments use a single model (GPT-5.4-mini) and runtime (Codex CLI); generalization to other model families/runtimes is not demonstrated.
  • The MHTBA code harness had portability issues on TB2 that inflate the apparent NLAH advantage on that benchmark (discussed in Appendix C, not shown in excerpt).

Relevance to Harnesses / Meta-Harnesses

This paper is a direct foundational contribution to the harnesses/meta-harnesses research thread: it operationalizes the concept of a "harness" as a first-class scientific object—inspectable, transferable, and ablatable—by separating harness policy into NLAHs and harness mechanism into a shared runtime (IHR). The paper explicitly cites Meta-Harness (Lee et al., 2026) as a related approach that automatically debugs and optimizes executable code harnesses, positioning NLAH+IHR as a complementary direction that trades hard code control for interpretability and modularity. The module ablation framework (Table 5) is particularly valuable for meta-harness research: it provides a controlled methodology for attributing performance changes to specific harness design choices (state management, evidence discipline, search branching) under a shared execution substrate, something that is practically impossible when harness logic is entangled in controller code.