Skip to content

Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

🕒 Published (v1): 2026-07-14 21:39 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Harness Handbook is a behavior-centric representation of agent harness codebases, built via static program analysis and LLM-assisted structuring, that organizes implementation knowledge around runtime behaviors rather than files/functions. It introduces Behavior-Guided Progressive Disclosure (BGPD) to guide coding agents from high-level behavior descriptions to verified source locations. Experiments on two open-source harnesses show improved localization accuracy and edit-plan quality while reducing planner token consumption.

Problem

Production agent harnesses are large, tightly coupled systems where a single behavior is distributed across multiple files, execution stages, and shared states. When a developer or coding agent needs to modify a behavior, they must first locate all relevant implementation sites — a step existing tools (repository maps, code search, long-context editing) fail to support because they organize knowledge around files and modules, not runtime behaviors. This behavior localization bottleneck is the central gap addressed.

Method

Harness Handbook is a three-level document structure (\(L1\)–\(L3\)) plus a cross-stage state-register view \(Z\): - L1: System overview (architecture, execution model, global data flow) - L2: Stage-level component cards (responsibilities, I/O, dependencies) - L3: Unit deep-dives with source-grounded locators (file path, function/region anchor, code excerpt)

Construction runs in three phases: 1. Static Fact Extraction (deterministic, no LLM): parse repository into a program graph \(G\) of functions, call edges, and state read/write accesses. 2. Behavioral Organization: map source units to execution-stage skeleton \(S\) via an iterative propose-review loop (function-as-leaf mode, requires a seed skeleton) or file-level summarization + inferred skeleton (file-as-leaf mode). 3. Hierarchical Synthesis: generate the \(L1\)–\(L3\) tree, validate every \(L3\) locator against the live repository, and package for localization.

BGPD then guides modification: start at \(L1/L2\) to identify relevant stages, follow \(Z\) to capture state-coupled stages, expand along the call graph, resolve candidate locators against the current repository, and return verified evidence \(E^b_q\) for edit planning. After execution, any non-empty diff \(\Delta\) triggers incremental resynchronization (fingerprint-based function matching; only affected entries rebuilt unless the stage skeleton is invalidated).

Key Contributions

  • Formal definition of behavior localization as identifying all code locations implementing a behaviorally described modification request.
  • Harness Handbook: a behavior-centric, three-level (\(L1\)–\(L3\)) + state-register (\(Z\)) representation automatically built from existing codebases.
  • BGPD (Behavior-Guided Progressive Disclosure): a coarse-to-fine navigation workflow that verifies candidate locations against the live repository before planning.
  • Automated construction pipeline with two leaf modes (function-as-leaf, file-as-leaf) and incremental resynchronization after each diff.
  • Evaluation benchmark: 60 behavior-driven modification requests (30 per harness) categorized by type (Query, Cross-file, Search-Hostile) and localization difficulty (Easy/Medium/Hard).

Results

Plan quality judged by three independent LLMs (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) on a weighted score \(S = 0.5 S_{\text{Loc}} + 0.25 S_{\text{Scope}} + 0.25 S_{\text{Reason}}\):

  • Overall win rate gains (Handbook-Assisted vs. Baseline):
  • Codex harness: +18.9 pp overall
  • Terminus-2 harness: +10.0 pp overall
  • Per-judge win rate gains (Codex / Terminus-2 respectively):
  • GPT-5.5: +26.7 / +13.3 pp
  • Opus 4.8: +10.0 / +10.0 pp
  • DeepSeek-V4-Pro: +16.7 / +10.0 pp
  • Localization dimension gains (Codex / Terminus-2): up to +23.3 / +13.4 pp
  • Token cost reduction per request: −12.7% on Codex, −8.6% on Terminus-2
  • Largest gains on Cross-file and Search-Hostile request types, and Hard difficulty requests (scattered implementations, rarely executed paths, cross-module interactions).

Limitations

  • Evaluation covers only two open-source harnesses; generalizability to closed or highly heterogeneous codebases is unconfirmed.
  • The function-as-leaf mode requires a trustworthy seed skeleton — this prerequisite may not be satisfiable for arbitrary harnesses.
  • \(L3\) locators that cannot be revalidated are frozen and excluded, meaning coverage degrades silently after major refactors until manual resync.
  • Planning is evaluated in read-only mode (NexAU + DeepSeek-V4-Pro); actual execution quality and downstream task success rates are not reported.
  • LLM judge agreement and potential bias from using models also involved in construction (Opus 4.8) are not fully ablated.
  • No evaluation of resynchronization latency or fidelity under large structural diffs.

Relevance to Harnesses / Meta-Harnesses

Harness codebases are increasingly the locus of system-level intelligence — routing, scaffolding, tool dispatch, reflection loops — yet they resist standard code-navigation tools because their behavior is distributed across files and only meaningful at runtime. This paper directly attacks that problem by building a static-analysis + LLM pipeline that produces a behavior-indexed representation, letting a coding agent (or human) navigate from "what does the retry logic do" to the exact source lines without reading the whole codebase. The BGPD mechanism is particularly relevant for meta-harness work: as you layer orchestration over orchestration, the ability to programmatically locate and edit specific behavioral seams — without full re-comprehension — becomes the bottleneck for safe, incremental evolution. Concretely, if you're building or modifying a harness (e.g., swapping a planner, adding an observation hook), this approach could reduce the cost of understanding where to make the change and increase confidence that the edit is complete and correct.