Skip to content

Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning

🕒 Published (v1): 2026-07-05 22:11 UTC · Source: Arxiv · link

Why this paper was selected

Frames agent harness as learnable via offline RL — directly novel for harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper reframes the LLM agent harness — the execution control layer that sequences OBSERVE, RETRIEVE, DRAFT, CHECK, REVISE, and SUBMIT operations — as a learnable policy rather than fixed infrastructure. A lightweight MLP controller is trained via offline advantage-weighted regression on terminal task-rubric rewards, keeping the frozen LLM executor unchanged. The learned controller reliably increases verification-before-submission behavior and selectively improves final task quality where the offline buffer contains high-return trajectories.

Problem

Agent harnesses are nearly always static artifacts: hand-written rules, fixed execution graphs, or prompt templates that apply identical control logic regardless of task difficulty, trajectory state, or intermediate outcome. A rule like "always check" wastes budget when a solution is already complete; "submit after drafting" fails when the draft is unverified. The failure mode is poor external control sequencing, not missing model knowledge, yet the harness has been treated as inert infrastructure rather than an optimizable policy.

Method

Harness MDP formulation. The authors define a finite-horizon MDP where the state \(s_t\) encodes trajectory progress, draft availability, evidence coverage, tool outputs, verifier feedback, recent failure count, remaining budget, and the previous action. The action space \(\mathcal{A} = \{\text{OBSERVE}, \text{RETRIEVE}, \text{CALL-TOOL}, \text{DRAFT}, \text{CHECK}, \text{REVISE}, \text{SUBMIT}\}\) operates over execution structure; the LLM parameters and prompts are never updated.

Offline advantage-weighted regression (AW). A finite rollout buffer \(\mathcal{B}\) is collected from base and exploratory harnesses. Terminal rubric scores define trajectory advantages \(A(\tau)\). The controller is trained by: $\(\mathcal{L}(\theta) = -\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\exp\!\bigl(A(s,a)/\beta\bigr) \cdot \log \pi_\theta(a \mid s)\right]\)$ with \(\beta=0.2\) and clipped weights in \([0.1, 10.0]\). No action-pattern bonuses are used; only terminal task-rubric reward.

Controller architecture. A one-hidden-layer MLP (64 units, softmax output) with invalid actions masked by domain adapters. Training uses Adam (\(\text{lr}=10^{-3}\), batch 256, 20 epochs, entropy coefficient 0.01), three seeds per domain, fixed hyperparameters across all domains.

Harness Maturity Score (HMS). A normalized weighted diagnostic over seven process events — CheckBeforeSubmit, EvidenceBeforeClaim, TestBeforeSubmit, RevisionAfterFailure, ValidToolUse, StopWhenSufficient, EarlySubmit (penalty) — evaluated post-hoc but never optimized directly.

Theoretical grounding. Theorem 1 proves optimal-policy invariance under potential-based reward shaping (the paper uses \(\Phi_t \equiv 0\), i.e., pure terminal reward). Theorem 2 gives a finite-buffer characterization: the outcome improvement is bounded by the buffer support ceiling \(G^\star_\mathcal{B}\), while process shifts are governed by \(\Delta\Psi_\mathcal{B} = \text{Cov}_{\mu_\mathcal{B}}(w, \Psi) / \mathbb{E}_{\mu_\mathcal{B}}[w]\), decoupling process learning from outcome learning.

Key Contributions

  • Formalizes LLM agent harness control as a Harness MDP with a compact structural action space, separating executor and controller.
  • Derives an offline RL training objective (advantage-weighted regression) for harness controllers using only terminal task-rubric rewards, with no LLM fine-tuning.
  • Introduces the Harness Maturity Score (HMS) as a process diagnostic separate from task-outcome reward, with a formal justification for why optimizing action-pattern bonuses can corrupt the task-optimal policy (Proposition 1).
  • Provides a finite-buffer support analysis (Theorem 2, Corollary 1) that analytically separates when process behavior improves from when final quality improves.
  • Evaluates across six controlled domains (100 tasks each) and two public-benchmark adapters (\(\tau\)-bench retail, AgentBench DB-Bench), with ablations against behavior cloning (BC) and Forced CHECK (FC).

Results

  • CheckBeforeSubmit increases from 0% baseline to 5.6–17.8% across all six controlled domains and 16.7–17.2% on both adapters.
  • \(\tau\)-bench retail adapter: \(\Delta G = +18.2\) pp (0.337 → 0.519), 95% CI [+15.1, +21.3].
  • AgentBench DB-Bench adapter: \(\Delta G = +13.2\) pp (0.415 → 0.547), 95% CI [+10.2, +16.2].
  • Coding (calibrated verifier): \(\Delta G = +10.0\) pp (0.712 → 0.812), 95% CI [+5.9, +14.6].
  • HMS improves in 5 of 6 controlled domains; research is the exception (EarlySubmit offsets CheckBeforeSubmit gains).
  • AW vs. BC: AW outperforms BC in all 8 settings; e.g., coding AW +10.0 pp vs. BC −8.3 pp.
  • AW vs. Forced CHECK: AW outperforms FC in 5 of 8 settings, ruling out mechanical verification as the explanation.
  • Knowledge-work (+1.4 pp), planning (+2.6 pp), multi-tool (−1.3 pp), long-memory (−0.3 pp) show minimal or null final-quality gains.

Limitations

  • Final-quality improvement is hard-capped by \(G^\star_\mathcal{B}\) (the best return in the offline buffer); the method cannot recover from a buffer lacking strong task-solving trajectories.
  • Adapter results use the authors' Harness MDP scoring protocol, not official \(\tau\)-bench or AgentBench scoring; gains cannot be directly compared to published leaderboard numbers.
  • Coding gain of +10.0 pp depends on the calibrated structural verifier; under the original strict deterministic rubric, the base harness was near saturation (ceiling effect).
  • Held-out evaluation sets are small (20 tasks per domain; 20 tasks per adapter), so bootstrap intervals on \(\Delta G\) are wide.
  • HMS gains are concentrated in CheckBeforeSubmit; most other process events (EvidenceBeforeClaim, RevisionAfterFailure, ValidToolUse) remain sparse or saturated.
  • No online exploration: the method cannot close gaps that require trajectories absent from the offline buffer.

Relevance to Harnesses / Meta-Harnesses

This paper directly operationalizes the harness as a first-class learnable object rather than fixed scaffolding, making it one of the most explicit treatments of harness-layer optimization in the literature. The Harness MDP formalism provides a reusable abstraction for thinking about any multi-step agentic execution pipeline — OBSERVE / RETRIEVE / DRAFT / CHECK / REVISE / SUBMIT can map cleanly onto the control decisions in existing meta-harnesses. The HMS metric fills a gap in harness evaluation: most harness work measures only final task outcomes, whereas HMS gives a process-level diagnostic that can identify whether a harness evolves sound execution habits independently of whether those habits improve scores. The finite-buffer support theorem (Theorem 2) is directly actionable for anyone building offline or semi-offline harness-learning systems, as it analytically separates the conditions under which process-level improvements translate to outcome-level gains.