Skip to content

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

🕒 Published (v1): 2026-05-27 03:47 UTC · Source: Arxiv · link

Why this paper was selected

Harness-Bench directly benchmarks harness effects on model outputs; fills critical measurement gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Harness-Bench is a diagnostic benchmark that isolates the harness — the execution layer managing context, tools, state, permissions, and recovery — as the primary axis of evaluation across shared task environments and model backends. Across 5,194 trajectories over 106 sandboxed tasks, it shows a 23.8-point performance gap between harness configurations under the same models, demonstrating that agent capability cannot be attributed to the base model alone. The benchmark further identifies recurring execution-alignment failures where plausible model reasoning decouples from tool feedback, workspace state, or verifiable output contracts.

Problem

Existing agent benchmarks (SWE-bench, AgentBench, GAIA, WebArena) either abstract away execution, conflate harness with the full agent stack, or hold the harness fixed when comparing models. This makes it impossible to study how the execution layer — prompt templates, action formats, context construction, tool invocation, budget control, tracing, and recovery — affects completion, cost, safety, and robustness independently of the underlying model.

Method

Harness-Bench fixes external task conditions (task prompt, initial sandbox state, budget, timeout, evaluator) while varying harness configuration and model backend in a full factorial design. The evaluation identity is \(\text{Agent} = \text{Model} + \text{Harness}\), with runs formalized as \(R = \text{Run}(M, H, E, T)\) and \(\text{TaskScore} = \text{Eval}(R; J)\).

Each task runs in an isolated offline sandbox. Scoring uses a multiplicative formula: $\(\text{TaskScore}_i = \text{Security}_i \cdot \text{Completion}_i \cdot \text{Process}_i\)$ where \(\text{Security}_i \in \{0,1\}\) gates on permission/constraint violations, and $\(\text{Process}_i = \frac{\text{Robustness}_i + \text{ToolUse}_i + \text{Consistency}_i}{3}\)$ All non-binary scores are normalized to \([0,1]\). LLM-based process rubrics use claude-sonnet-4.6 as a fixed external judge. Each run records final workspace state, execution trace, usage statistics, and validator outputs.

Key Contributions

  • Harness-Bench task suite: 106 manually reviewed, sandboxed offline tasks across 8 workflow categories (software engineering, data/BI analytics, workspace/tool operations, knowledge/evidence retrieval, office workflows, vertical professional workflows, long-running autonomy, SRE/DevOps), each with fixtures, evaluator, and manifest.
  • Model–harness evaluation protocol: fixes external task conditions while preserving each harness's native execution behavior; 6 configurable harnesses × 8 API model backends = 5,088 trajectories, plus 106 from Codex, totaling 5,194.
  • Diagnostic failure taxonomy: five recurring execution-alignment failure symptom categories identified from trajectory analysis (contract/format, tool/recovery, evidence/grounding, artifact commitment, state/continuation).

Results

  • Among configurable harnesses (averaged over 106 tasks and 8 model backends): NanoBot 76.2%, Hermes 71.2%, Moltis 68.8%, NullClaw 64.4%, ZeroClaw 61.4%, OpenClaw 52.4% — a 23.8-point spread.
  • Codex (model-bound, GPT-5.4 backend): 80.4% aggregate, 86.5% completion — reported separately as it is not a configurable harness.
  • All harnesses achieve 100% security score under the Harness-Bench protocol.
  • NanoBot achieves the highest configurable-harness score while using only 68.7K tokens/run; NullClaw scores lower (64.4%) but uses 175.1K tokens — longer trajectories do not determine performance.
  • Stronger model backends achieve higher mean scores and lower cross-harness variance, indicating they are more tolerant of harness differences; weaker backends show higher variance.
  • Recurring failure symptoms in failed trajectories: contract/format violations 36.4%, tool/recovery failures 24.6%, evidence/grounding gaps 14.6%, artifact commitment failures 11.1%, state/continuation failures 9.3%.

Limitations

  • Evaluates complete harness configurations, not individual harness mechanisms — causal decomposition of which harness component drives differences is not supported.
  • Process scores include LLM-assisted rubric assessment, introducing judge model dependency (fixed to claude-sonnet-4.6).
  • Results are descriptive measurements over a fixed 106-task suite; no statistical claims about generalization to all agent workflows.
  • All tasks are offline/sandboxed; findings may not fully transfer to live-service or drifting-state settings.
  • 8 model backends and 6 harnesses form a specific sample; coverage of the full harness and model space is limited.

Relevance to Harnesses / Meta-Harnesses

Harness-Bench directly operationalizes the harness concept — the system layer mediating between model outputs and external actions — as a first-class evaluation axis, which is the core object of interest for harness and meta-harness research. The benchmark provides empirical evidence that harness choice causes performance variation on the order of 24 percentage points under identical task conditions, substantiating the engineering importance of execution-layer design choices (context policy, tool interface, recovery, budget control, tracing). The execution-alignment failure taxonomy (especially contract/format and artifact commitment failures) points to concrete harness responsibilities that meta-harnesses must handle. The finding that stronger models exhibit lower cross-harness variance has direct implications for meta-harness design: a meta-harness routing across multiple sub-harnesses or models can reduce sensitivity to any single harness's idiosyncrasies by preferring stronger backends.