MemoHarness: Agent Harnesses That Learn from Experience

🕒 Published (v1): 2026-07-14 21:22 UTC · Source: Arxiv · link

Why this paper was selected

Literally defines/studies harness layer; memory-augmented harness design; most on-topic

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MemoHarness is an adaptive agent harness optimization framework that decomposes the harness into six editable control dimensions, accumulates per-case diagnoses and distilled global patterns in a dual-layer experience bank during training-time search, and at test time retrieves relevant experience to specialize the global harness to each unlabeled case—without any test-time feedback or gradient updates. On Terminal-Bench (shell agency), it reaches 0.806 versus 0.722 for the strongest fixed-harness baseline, with selective positive transfer to unseen suites and base models.

Problem

Most automatic agent improvement methods optimize narrow artifacts—prompts, pipelines, or workflows—rather than the full harness control layer (context assembly, tool interfaces, decoding, orchestration, memory, output handling). The one prior system treating harness code as the search object (Meta-Harness, Lee et al. 2026) produces a single training-time artifact reused for all cases; no existing approach performs per-case, test-time harness specialization without labels or additional search.

Method

MemoHarness operates in two phases over a six-dimensional harness space \(\mathcal{W} = \mathcal{W}^{(1)} \times \cdots \times \mathcal{W}^{(6)}\) (context assembly, tool/retrieval, generation, orchestration, memory, output handling).

Phase A — Training-time search. Starting from a minimal harness \(W_0\) (no demos, deterministic single-call, raw output passthrough), a controller iteratively proposes candidates \(W_t\), executes each on labeled search cases \(\mathcal{D}^\text{search}\), scores by a correctness-first reward \(r_i(W)\) with token usage \(c_i(W) = n_i^\text{tok}\) as tiebreaker, and writes a per-case execution entry \(\xi_i^{(t)} = (i, t, \phi_i, W_t, \Delta_i^{(t)}, \tau_i(W_t), r_i, c_i, z_i^{(t)})\) to the experience bank. A diagnostic operator \(z_i^{(t)} = g(x_i^s, W_t, \tau_i, r_i)\) emits a primary failure dimension \(d_{i,\text{prim}} \in \{1,\ldots,6\} \cup \{\emptyset\}\) plus a natural-language analysis. Every \(N\) iterations, a distillation operator extracts persistent cross-case patterns \(G_t\) from failure clusters. The final global harness is selected by lexicographic ordering: \(W^\star \in \arg\max_{\text{lex}} (\bar{r}_t, -\bar{c}_t)\).

Phase B — Test-time case adaptation. For each unlabeled test case \((u_j, \phi_j)\), the controller retrieves top-\(K\) successful and failed historical entries by cosine similarity \(\rho_\psi(x, \xi) = \cos(\psi(u), \psi(u_\xi))\), plus feature-conditioned global patterns, then emits a case-specific harness \(W(x_j) = \Pi^\text{test}(W^\star, x_j, \mathcal{S}^\text{test}(x_j))\) executed without any iterative feedback.

Key Contributions

• Six-dimensional structured harness space decomposing the inference pipeline into separable, targetable control surfaces (D1–D6).
• Dual-layer experience bank \(\mathcal{B}_t = (\mathcal{E}_t, \mathcal{G}_t)\) that stores per-case execution entries with dimension-level failure attribution alongside periodically distilled global patterns, making search diagnostic rather than purely score-driven.
• Test-time case adaptation via nearest-neighbor retrieval over the frozen bank, specializing the global harness to each unseen case without labels, re-search, or gradient updates.
• Correctness-first selection rule (lexicographic: reward primary, token cost secondary) that prevents cost-driven drift toward cheap-but-wrong configurations.
• Empirical evidence of selective positive cross-dataset and cross-model transfer without retraining.

Results

• Terminal-Bench (shell agency): MemoHarness 0.806 vs. strongest baseline Codex 0.722 (+0.084); improvements over other baselines range +0.250 to +0.445.
• LiveCodeBench (code generation): base 0.900 → final 0.967; near-ceiling regime, gain is modest.
• FinanceAgent (financial reasoning): base 0.600 → final 0.767; peak 65.0% at iterations 8–9 (from 42.5% base).
• Cross-dataset transfer (Table 2, GPT-5.3-Codex, 6 external suites): Terminal-Bench harness improves MMMLU +0.030, StrongReject +0.030, SWE-Bench Pro +0.059; saturated suites (HumanEvalFix, Reasoning-Gym-Easy) show no movement.
• Cross-model transfer (Table 3, 6 additional models): mean gain +0.098 over base setting per model, ranging +0.038 (GPT-4.1) to +0.233 (GLM-5).
• Retrieved context can be cached, described as cost-competitive relative to commercial baselines, though detailed cost table (Table 4) is cut off in the provided text.

Limitations

• Statistical robustness and component attribution are explicitly deferred to future work; the paper does not report confidence intervals or error bars.
• Cross-dataset transfer is selective, not uniform; some source–target pairings show no improvement or mixed results (e.g., LawBench).
• Cross-model transfer cannot be decomposed into model-specific vs. harness-specific effects without additional ablations.
• Baselines involving product-specific systems (e.g., Codex) are compared under a shared evaluation protocol, not pure scaffold-only conditions, complicating isolation of harness contribution.
• The diagnostic operator and distillation operator are LLM-driven; their reliability and failure modes are not analyzed.
• Test-time adaptation latency and caching behavior are only partially characterized (Table 4 truncated).

Relevance to Harnesses / Meta-Harnesses

MemoHarness is directly on-topic: it treats the full agent harness as a first-class optimization target—not just prompts or pipelines—and positions itself as an extension of the Meta-Harness line (Lee et al. 2026) by adding test-time, per-case adaptation over a structured six-dimensional harness space. The dual-layer experience bank (per-case entries + distilled global patterns) is a new mechanism for making harness search accumulate reusable diagnostic knowledge, which is a qualitative step beyond single-artifact optimization. For researchers tracking harness/meta-harness work, the key advance is the separation of global harness search from instance-level specialization at deployment, and the empirical evidence that execution experience—rather than gradient signal—is a practical substrate for adaptive harness design.