Skip to content

Dissecting model behavior through agent trajectories

🕒 Published (v1): 2026-06-16 03:17 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

The paper formalizes the intent-execution gap — a bidirectional mismatch between what a model intends and what the harness executes (and vice versa) — as a first-class design concern in agent systems. The authors build Simple Strands Agent (SSA), a single open-source harness with minimal per-family adapters, that reproduces or exceeds official pass@1 numbers for 21 models across 5 families on SWE-Bench-Verified, SWE-Bench-Pro, and Terminal-Bench-2. Beyond outcome metrics, they analyze 138k SSA trajectories to surface stable behavioral signatures invisible to aggregate pass@1.

Problem

Existing agent harnesses are either bespoke (over-fit to one model/benchmark) or ignore the gap between model intent and harness execution. As a result, a more capable model \(M_A > M_B\) can yield \(\text{acc}(H, M_A) \approx \text{acc}(H, M_B)\) under a misaligned harness \(H\). Additionally, pass@1 collapses all trajectory variation into a binary signal, hiding how different model families allocate effort, backtrack, or transition between exploration and implementation phases.

Method

Simple Strands Agent (SSA) is structured around four design principles:

  1. Feedback-rich tool boundaries: shell calls return command, status, exit code, stdout, stderr, and partial output on timeout; edits are anchored to neighboring lines and return a unified diff on success; partial/duplicate matches are rejected with actionable feedback.

  2. Reasoning nudge: a prompt-level balance between chain-of-thought and tool-calling. For Claude variants, a quantitative target ("make 50+ tool calls") breaks long reasoning chains; for Gemini/Grok, a flexible directive ("use tools as much as possible") is used instead. The principle is model-evidenced action over internal assumption.

  3. Model-specific tool-surface alignment: GPT models get apply_patch-style editing; Grok gets atomic (split) edit/view tools and line-number display disabled — matching learned tool-call preferences from training.

  4. Solution distance \(D_i(t)\): a recall-divergence from the agent's current repository state to the nearest oracle-correct patch in a text-level code space \(\mathcal{X}\). Formally, each code state \(x\) maps file paths to exact text strings; the divergence is:

\[D_i(t) = \min_{x^\star \in S_i} \left(1 - \frac{|\varphi_i(x_i(t)) \cap \varphi_i(x^\star)|}{|\varphi_i(x^\star)|}\right)\]

where \(\varphi_i\) is a unified-diff patch-feature projection and \(S_i\) is the oracle-correct solution set. \(D\) starts at 1 and falls to 0 when the current state reproduces a verified fix; \(\Delta D > 0\) signals backtracking.

Key Contributions

  • SSA harness: single open-source harness reproducing or exceeding official pass@1 for 21 models across 5 families (Claude, GPT, Gemini, Grok, Qwen) on three agentic benchmarks.
  • Solution-distance metric over a text-level code state space enabling per-edit, per-trajectory quality analysis beyond pass@1.
  • Trajectory-level behavioral analysis of 138k trajectories revealing family-specific signatures (edit/test ratios, phase schedules, backtracking rates) stable across benchmarks and invisible to aggregate metrics.
  • Git-history leakage discovery: public SWE-Bench-Pro containers retain reachable future git objects, inflating measured pass@1 by up to 6.9% for affected models.

Results

  • SWE-Bench-Pro (n=731): pass@1 ranges 27.4% (Qwen3-Coder 30B) to 59.9% (GPT-5.4); Opus 4.6 at 57.45%, Qwen3-Coder Next at 57.75%, GPT-5.3 at 58.57%; pass@2 peaks at 69.8% (Qwen3-Coder Next).
  • SWE-Bench-Verified (n=500): pass@1 ranges 51.4% to 80.8%; Opus 4.6 and Gemini 3.1 Pro tied at 80.80%; Gemini 3.1 Pro leads pass@2 at 84.1%.
  • Terminal-Bench-2 (n=89, constrained setting): GPT-5.3 75.05%, Opus 4.6 73.93%, GPT-5.4 72.58%, Gemini 3.1 Pro 72.13%.
  • All official model-card numbers fall within SSA's 95% CI (single exception: GPT-5.2-codex on SWE-Bench-Pro, confounded by leakage).
  • SSA outperforms widely-used open-source harnesses under identical model configurations.

Limitations

  • \(S_i\) (oracle-correct solution set) is not enumerable in practice; \(D_i(t)\) is approximated with a finite empirical reference set of verified patches.
  • Family-specific adapters (prompt nudges, tool-surface choices) are still required — SSA is not fully universal.
  • SWE-Bench-Pro comparisons remain partially confounded by container leakage even after auditing; cross-model comparisons at small margins carry uncertainty.
  • Terminal-Bench-2 results are infrastructure-sensitive (memory/time limits), requiring separate constrained/unconstrained reporting, making fair comparison harder.
  • The paper's text is truncated before full Terminal-Bench-2 and trajectory-analysis sections are reported.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct contribution to harness design theory: it names and formalizes the intent-execution gap as a core design primitive, placing harness-model alignment on equal footing with model capability and tool selection. SSA is a reusable, multi-family harness that demonstrates how invariant interface principles (structured feedback, safe editing semantics, reasoning nudges) can generalize across model generations without per-benchmark over-fitting — exactly the problem meta-harness research aims to solve. The solution-distance metric provides a harness-agnostic instrumentation layer for trajectory-level quality assessment, which is a missing piece in current harness evaluation methodology. The git-history leakage finding also serves as a cautionary result about benchmark integrity in harness-driven evaluation pipelines.