Skip to content

HARBOR: Automated Harness Optimization

🕒 Published (v1): 2026-04-22 13:45 UTC · Source: Arxiv · link

Why this paper was selected

First paper automating harness optimization end-to-end; directly on topic

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HARBOR frames agent-harness configuration as a constrained noisy Bayesian optimization problem over a mixed-variable, cost-heterogeneous flag space. A reference solver—built on a block-additive SAAS surrogate, multi-fidelity cost-aware acquisition, and TuRBO trust regions—is proposed to replace manual tuning. A four-round manual tuning case study on a production coding agent shows that stacking published harness features is net-negative, motivating automation.

Problem

Long-horizon LLM agent harnesses accumulate dozens of flag-gated features (context compaction, caching, speculative execution, self-evaluation, cross-session memory). Manual ablation is lossy: it evaluates configurations sequentially, misses cross-flag interactions, cannot detect silent integration bugs (zero-counter flags), and cannot systematically explore a space where \(|C| \gg 2^{40}\). No prior work treats harness configuration (as opposed to harness code generation) as a formal optimization problem.

Method

Configuration space. The harness is modeled as \(\mathcal{C} = \prod_{i=1}^{F} C_i\), where each \(C_i\) is Boolean, numeric, or a discrete preset; \(F \approx 40\) nominal flags, effective \(F_{\text{eff}} \approx 5\) after Axis-IV silent-flag auto-exclusion.

Reward. Cold-start-corrected pass rate using Wilson score intervals; warm-start-aware evaluator (Eq. 3) accounts for cross-session features that require prior task exposure to produce signal.

Surrogate. Block-additive SAAS (Sparse Axis-Aligned Subspace) GP that tolerates the high ambient dimension by shrinking irrelevant axes; updated in-place as Axis-IV prunes zero-signal flags.

Acquisition. Knowledge Gradient divided by cost, \(\text{KG}/c(m)\), with multi-fidelity evaluation levels \(m \in \mathcal{M}\); cheap low-fidelity runs screen candidates before expensive full-suite evaluations.

Trust regions. TuRBO restricts BO proposals to local trust regions to prevent global pathologies in high-dimensional discrete-continuous spaces.

Safety check. Posterior chance-constrained filter: a proposed configuration is only deployed if the posterior probability that its pass rate exceeds a minimum threshold exceeds a confidence level; outputs a Pareto front on (pass-rate, cost).

Runtime telemetry. Write-side vs. consumer-side counters expose integration bugs (flag requested but never consumed) as invariant violations, feeding Axis-IV auto-exclusion.

Key Contributions

  • Formalization of Automated Harness Optimization (AHO) as constrained noisy BO over a mixed-variable, cost-heterogeneous configuration space with cold-start-corrected rewards and posterior chance-constrained safety.
  • HARBOR reference algorithm: block-additive SAAS surrogate + \(\text{KG}/c\) multi-fidelity acquisition + TuRBO trust regions + Axis-IV silent-flag auto-exclusion.
  • A frankly negative four-round manual tuning case study (codex-py / Terminal-Bench 2) demonstrating that literature-grounded stacking produces at most one credible net win and that two silent integration bugs went undetected across all rounds.
  • Warm-start-aware evaluator using per-flag telemetry counters that can surface zero-consumer bugs as observable invariant violations.

Results

  • Baseline (all-flags-off): 15/89 on Terminal-Bench 2.
  • Round B (best manual): 17/89 (+2; cache-key normalization with mtime-indexed keys, failure-aware trajectory library, adaptive hint thresholds); cache hit rate rose from ~1% to 3.7% session-weighted / ~12% intra-session.
  • Round C (+ self-evaluation gate, canonicalization, prompt-prefix layout): 13/89 (−4 from B); self-evaluation gate corrected passing answers to failing ones under gpt-5.4-nano; intent-canonicalization produced zero canonical cache hits.
  • Round D (+ ACON, Reflexion, PASTE): 12/89 (−5 from B); Reflexion retrieval counter = 0 (cross-session store not propagated across containers); PASTE prediction counter = 0 (predictor entry-point never wired into runner).
  • Oracle (best-of-all-configs union): 81/89—demonstrating 64 tasks are solvable by some configuration but not by the single best manual configuration.
  • Meta-Harness (cited prior work) reports +7.7 absolute accuracy points on LawBench with 4× fewer context tokens vs. hand-engineered baselines.

Limitations

  • Single benchmark (Terminal-Bench 2) and single agent (codex-py); authors explicitly disclaim conference-grade universality claims.
  • End-to-end HARBOR results are reported but concrete model/benchmark identities are deferred to §VII (truncated in the provided text); quantitative BO vs. manual comparison numbers are not fully visible.
  • Cold-start correction applies only when a warm-start-capable task suite is available; benchmarks like Terminal-Bench 2 are task-independent by design, amplifying the cold-start penalty for cross-session features.
  • \(F_{\text{eff}}\) collapse by Axis-IV assumes silent flags are identifiable via telemetry counters—requires instrumented harnesses; un-instrumented harnesses cannot benefit.
  • Binary embeddings and Matryoshka-style two-stage retrieval noted as natural next steps but not evaluated.

Relevance to Harnesses / Meta-Harnesses

HARBOR is among the first papers to treat harness configuration—not code generation—as a formal ML optimization problem, establishing a clean conceptual boundary between Meta-Harness-style LLM-driven harness code rewriting and BO-driven hyperparameter search over a fixed flag space. The negative case study directly quantifies the failure modes of manual harness stacking (silent integration bugs, non-additive interactions, cold-start crippling) that any meta-harness system must either avoid or detect. The AHO formalism—mixed-variable configuration space, cold-start-corrected reward, multi-fidelity cost-aware acquisition, posterior safety check—is a reusable scaffold applicable to any agent harness with a bounded flag space, making it directly actionable for researchers building or auditing harnesses at scale. The Axis-IV telemetry pattern (write-side vs. consumer-side counters as invariant violation detectors) is a concrete observability recipe transferable to meta-harness evaluation loops.