Skip to content

Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

🕒 Published (v1): 2026-07-02 17:08 UTC · Source: Arxiv · link

Why this paper was selected

Observational study isolating reasoning effort vs tool access in agentic code generation reliability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

An observational study of 90 agentic coding runs building the same application under controlled conditions finds that reasoning effort — not tool access — is the lever that buys first-try reliability. Raising effort from High to xHigh on Claude Opus 4.7 lifted perfect-first-try runs from 28% to 89% at 9–29% extra cost, while adding a Playwright browser-testing tool raised cost 42–68% with zero functional improvement. The practical upshot: match the fix to the failure mode rather than adding capability indiscriminately.

Problem

Public agentic-coding benchmarks use single runs with a single instruction, conflating model choice with tool, prompt, and effort settings and hiding run-to-run variance. There is no replicated, matched evidence for whether common capability additions (browser-testing tools, design-oriented prompts, higher reasoning effort) actually deliver the improvements they are assumed to provide, or whether the cost is justified.

Method

90 independent agent runs each built a real-time retrospective board from one fixed OpenSpec specification. Every run was scored against a 14-criterion functional rubric (42-point maximum; criterion scores of 3 = first try, 2 = fixed after one corrective prompt, 1 = unresolved). Visual quality was rated 1–5 from screenshots by an LLM rater verified by a human. Two harnesses were used: Claude Code v2.1.132 (using /opsx:apply) and Antigravity (Google's agent). Within Claude Code runs, three factors were varied one at a time: Playwright tool availability, a design-oriented system prompt (full third-party vs. a one-paragraph abridged paraphrase vs. none), and reasoning effort (High vs. xHigh). Six repeated runs per cell in the main Opus 4.7 2×4 grid. Session token counts, cost estimates, lines of code, and model time were preserved from the harness's end-of-session report for decomposition analysis. Statistics used Fisher's exact test descriptively (not as causal identification).

Key Contributions

  • First matched, replicated contrast of a browser-automation testing tool (Playwright) at two capability tiers, holding model and effort fixed.
  • First High-vs-xHigh reasoning effort contrast in this dataset, showing a 28% → 89% first-try-perfect lift.
  • Separation of design-prompt effect on visual quality from its effect on function, with a prompt-length ablation.
  • Criterion-level failure profile across 90 runs, revealing that environment faults (Docker, local dev) account for 55% of first-try failures and shift across model generations while aggregate scores move less than one point.
  • Token-level cost decomposition showing the testing tool premium goes to cache re-reading (2×–3× cache-read tokens), not additional code output.

Results

  • Model tier: Frontier families (Opus 4.7, Opus 4.6, Sonnet 4.6) clustered at mean ~40.7–41.3/42; Qwen scored 37 and 24, ~10 points below frontier despite estimated token costs 12–50× the frontier median.
  • Reasoning effort (Opus 4.7): xHigh vs. High first-try-perfect rate: 89% (16/18) vs. 28% (5/18), Fisher exact \(p < 0.001\); corrective prompts reduced ~5-fold (3 vs. 16 total); median cost premium only 9% for base condition, 21–29% with additional capabilities.
  • Playwright testing tool: Zero functional score change on Opus 4.7 at both effort levels; median cost +42% at High, +68% at xHigh; cache-read tokens rose from 2.3M to 5.3M (High) and 7.0M (xHigh) while output tokens were flat or slightly lower.
  • Design prompt: Full prompt raised visual quality from 3.0 to 4.5/5; abridged one-paragraph version matched at 4.7/5 — confirming the lift needs only invocation, not the full prompt length. Functional scores unchanged or slightly lower (drag-and-drop criterion first-try failures: 11/40 design-prompted runs vs. 5/45 without).
  • Docker deployment: Failed first try in 44% (40/90) of runs; Opus 4.6→4.7 shift improved Docker first-try pass from 5/26 to 18/24 (\(p < 0.001\)) but worsened local dev environment failures (1/26 → 10/24); xHigh cleared both (17/18 Docker, 18/18 local dev).

Limitations

  • Observational design: conditions were assigned, not randomized; Fisher \(p\)-values are descriptive, not causal.
  • Single task (one retrospective-board spec); generalizability to other task types is unverified.
  • Agent execution environment (Windows WSL vs. native Windows) was not separately recorded, potentially introducing environment-dependent scoring artifacts.
  • Qwen runs: only 2 runs; cost figures are token-volume estimates at hosted-model rates, not actual prices.
  • Playwright-tool vs. no-tool contrast is used vs. available; design choice was driven by all 23 tool-enabled runs actually invoking it, so no passive-tool baseline exists.
  • Antigravity harness runs (5 total) are kept separate from Claude Code runs, so the harness contrast is underpowered.
  • The abridged prompt ablation cells were added after the main sweep, making them not part of a pre-specified grid.

Relevance to Harnesses / Meta-Harnesses

This study is directly structured around two named agent harnesses — Claude Code and Antigravity — and treats harness configuration (system prompt injected via CLAUDE.md, tool enablement, effort flag) as the primary experimental variable, making it an empirical meta-harness evaluation. The finding that the Antigravity design prompt, extracted from that harness and replanted into Claude Code, reproduces the same visual-quality lift is a concrete demonstration of harness-level prompt portability and prompt-effect isolation — a key concern in meta-harness design. The token decomposition (cache-read vs. output tokens per configuration cell) provides a cost-attribution methodology that meta-harness authors can apply to audit where capability additions actually spend budget. The result that environment faults (build, container, package installation) dominate failures and are invisible to an interface-level checking tool sharpens the scope question every harness designer must answer: which failure classes is the harness's verification layer able to reach?