Skip to content

Scaling Laws for Agent Harnesses via Effective Feedback Compute

🕒 Published (v1): 2026-05-28 09:45 UTC · Source: Arxiv · link

Why this paper was selected

Scaling laws for agent harnesses via feedback compute; bridges test-time scaling to harness design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent harness performance is poorly predicted by raw compute (tokens, tool calls, cost); this paper introduces Effective Feedback Compute (EFC), a trace-level scalar that credits feedback only when it is informative, valid, non-redundant, and retained, and shows it follows a power-law scaling relationship with failure rate (\(R^2\) up to 0.99) across synthetic, code-execution, and real benchmark traces. Task-demand normalization (EFC/\(D_\text{task}\)) removes residual cross-task scale mismatch and yields the strongest predictor. A matched-budget intervention directly demonstrates that improving feedback quality (holding raw cost fixed) raises success from 0.27 to 0.90.

Problem

Test-time scaling analyses for agent systems parameterize compute by raw quantities—tokens, tool calls, wall time, operations, cost—none of which distinguish useful feedback from redundant or unstable interaction. Two trajectories can consume identical raw budgets yet differ sharply in whether the feedback they generate is informative, grounded, non-redundant, and retained; existing coordinates therefore fail to predict harness failure rates reliably, leaving harness design without a principled scaling law.

Method

EFC definition. Each feedback event \(e_t\) in a trajectory \(\tau = \{(s_t, a_t, o_t, u_t)\}_{t=1}^T\) receives four bounded factors \(I_t, V_t, R_t, M_t \in [0,1]\) for informativeness, validity, non-redundant relevance, and memory update. The event contribution is: $\(\text{EFC}_t = \kappa I_t V_t R_t M_t\)$ with \(\kappa=10\), and run-level EFC sums over all feedback events.

Oracle vs. Estimated EFC. On synthetic controllable tasks, factors are assigned from hidden state (Oracle-EFC). For realistic traces, an event-level regression model \(\widehat{\text{EFC}}_t = \max(0, \exp(\theta_0 + \theta^\top \varphi(e_t)) - 1)\) is calibrated on synthetic oracle labels using nine trace-observable features (checker fires, tool-result reference, plan changes, memory retention, repeated-error avoidance, subgoal progress, etc.) and applied without final-outcome access.

NRS-EFC adds status-quality, progress-gate, and loop-type gates; it further discounts by attempt index to penalize redundant re-tries in noisy real traces.

Task demand normalization. \(D_\text{task} = L \cdot H_\text{tool} \cdot S_\text{state} \cdot (1+N_\text{obs}) \cdot (1-V_\text{oracle})\) scales EFC by minimum required steps, tool ambiguity, state-tracking demand, observation noise, and verifier coverage. Normalized \(X = \text{EFC}/D_\text{task}\) is the primary scaling coordinate.

Scaling model. A power-law failure model \(E(z) = E_\infty + Az^{-\alpha}\) is fit to each scalar coordinate; \(R^2\) and MAE over aggregated failure-rate cells are the primary metrics.

Harnesses evaluated. Seven families H0–H6 ranging from single-pass direct answer (H0) to deep closed-loop with structured memory and large interaction depth (H6), evaluated across DeepSeek-V4-Flash, gpt-5.4-nano, and Claude-Haiku-4.5 under budget sweeps. Three task layers: synthetic controllable (Needle Lookup, State Tracking, Rule Filter), semi-realistic executable (HumanEval-style), and real benchmarks (HumanEval, TerminalBench 2.0, SWE-bench Verified).

Key Contributions

  • Formalizes EFC as a multiplicative, trace-level measure of useful closed-loop feedback, with Estimated-EFC and NRS-EFC variants for settings without oracle state access.
  • Demonstrates that EFC and EFC/\(D_\text{task}\) outperform raw-compute baselines and the SAS multivariate baseline as scaling coordinates across controlled, executable, real, held-out, and prospective evaluations.
  • Decomposes harness scaling into two orthogonal factors: harness efficiency \(\eta = \text{EFC}/C_\text{raw}\) (raw-to-EFC conversion) and task demand \(D_\text{task}\) (sufficiency threshold), showing that both must be addressed for scaling gains.
  • Provides a matched-budget intervention isolating feedback quality causally from raw expenditure.

Results

  • Controlled synthetic tasks: Raw tokens \(R^2=0.33\), tool calls \(0.42\), SAS \(0.88\); Oracle-EFC and Estimated-EFC both \(0.94\); Oracle-EFC/\(D_\text{task}\) \(R^2=0.99\), MAE \(0.02\).
  • Matched-budget intervention: With raw cost delta = 0.000% and tool-call delta = 0, high-EFC condition raises success from 0.27 to 0.90 (\(p = 1.0 \times 10^{-300}\)).
  • Trace-time estimation (held-out tasks): Estimated-EFC/\(D_\text{task}\) reaches \(R^2=0.93\); Oracle-EFC/\(D_\text{task}\) reaches \(0.95\); raw tokens reach \(0.44\), SAS \(0.86\).
  • Executable code tasks (HumanEval-style held-out): Tool calls \(R^2=0.94\) (strong because they partially align with test feedback); Estimated-EFC/\(D_\text{task}\) \(0.97\); oracle variants \(0.99\).
  • Mixed real traces: NRS-EFC/\(D_\text{task}\) \(R^2=0.92\); raw compute has near-zero or negative fit.
  • Prospective holdout (prespecified metric and calibration): NRS-EFC/\(D_\text{task}\) \(R^2=0.85\), best among all predictors.
  • Module ablations: Harness efficiency \(\eta\) explains success variation with \(R^2=0.97\); raw cost explains almost none.
  • Efficiency is slice-dependent: Later harnesses dominate HumanEval-style tasks; all harnesses remain low-efficiency on Terminal tasks; SWE tasks favor earlier/mid-stage variants.

Limitations

  • Oracle-EFC requires hidden task state and is unavailable on purely real benchmarks; Estimated-EFC introduces calibration error and requires a feature-engineering step tied to the trace format.
  • EFC factors (\(I_t, V_t, R_t, M_t\)) are bounded scalars assigned by heuristics or regression proxies; the mapping from raw observations to these factors is not formally grounded and may not generalize across novel tool ecosystems or multi-agent topologies.
  • \(D_\text{task}\) is computed from hand-specified components (step count, tool ambiguity, state-tracking demand, noise, verifier visibility); estimating it for arbitrary real-world tasks without oracle knowledge is non-trivial.
  • The scaling law uses a fixed power-law form \(E(z) = E_\infty + Az^{-\alpha}\); regime changes (e.g., capability jumps in the base model) may violate this functional form.
  • Experiments use three relatively small/fast base models; whether the scaling coordinate is stable for frontier-scale models or across very different modalities is not tested.
  • The paper does not address how to optimize EFC during harness design, only how to measure it post-hoc or estimate it at trace time.

Relevance to Harnesses / Meta-Harnesses

This paper directly formalizes what makes an agent harness effective at a principled level: not raw expenditure but the quality of closed-loop feedback it extracts, retained in memory, and sufficient relative to task demand. For someone tracking harness and meta-harness design, EFC provides an actionable scalar that can serve as an optimization target—a meta-harness could in principle monitor \(\widehat{\text{EFC}}\) at trace time and dynamically re-route, retry, or escalate harness complexity when feedback quality is low. The harness efficiency decomposition (\(\eta = \text{EFC}/C_\text{raw}\)) also gives a principled vocabulary for comparing harness architectures (routing, verification, memory, observation quality) on equal footing, regardless of raw budget differences. The prospective validation result (\(R^2=0.85\) on a prespecified holdout) suggests EFC is robust enough to be used as a real-time signal within a meta-harness control loop.