Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

🕒 Published (v1): 2026-07-02 17:08 UTC · Source: Arxiv · link

Why this paper was selected

Empirical: reasoning effort beats tool access for agentic code reliability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This observational study ran 90 independent agentic coding sessions on a fixed task (a real-time retrospective board) under controlled conditions, varying model, reasoning effort, a browser-testing tool (Playwright), and design prompts. Raising reasoning effort from High to xHigh lifted first-try-perfect runs from 28% to 89% at 9–29% extra cost; the Playwright tool raised cost 42–68% with zero functional gain. The practical prescription: fix reasoning-level failures with more deliberation, not more tooling.

Problem

Public agent benchmarks typically run each model once on a short prompt, conflating model choice with tool and prompt configuration, and reporting post-repair totals that compress run-to-run variation and hide first-try reliability differences. The study targets this gap by holding the task constant and varying one factor at a time across replicated runs.

Method

Ninety graded runs of the same OpenSpec application spec were produced via two harnesses (Claude Code v2.1.132 and Antigravity). Five model families were tested: Claude Opus 4.7 (48 runs), Opus 4.6 (27), Sonnet 4.6 (10), Gemini 3.1 (3), Qwen (2). Within Claude Code, a full 2×4 grid was run for Opus 4.7: two reasoning effort levels (High, xHigh) × four prompt conditions (base, plus Playwright, plus full design prompt, plus one-paragraph abridged design prompt), six replications per cell. Each run was scored on a 14-criterion rubric: 3 points if the feature passed first try, 2 if repaired after one corrective prompt, 1 if unresolved; maximum 42. Visual quality was rated 1–5 from screenshots by an LLM rater, verified by a human. Token-level cost decomposition used end-of-session harness reports to attribute cost to output tokens vs. cache-read tokens.

Key Contributions

• First within-study, replicated effort contrast (High vs. xHigh) for an agentic coding task, isolating effort from model generation
• Matched, replicated contrast of a visual automation/testing tool (Playwright) holding model and effort fixed, at two capability tiers
• Criterion-level failure profile across 90 runs, decomposing where first-try failures cluster (environment vs. interface vs. data layer)
• Token-level cost decomposition distinguishing what each capability premium buys (cache re-reads vs. output generation)
• Prompt ablation showing a one-paragraph design directive reproduces the full visual quality lift of a lengthy third-party design prompt

Results

• Model tier dominates: Frontier families (Opus 4.7, 4.6, Sonnet 4.6) clustered near 41/42; Qwen scored 24–37, ~10 points below, at 12–50× the token volume
• Reasoning effort (main finding): High→xHigh on Opus 4.7 raised first-try-perfect runs from 28% (5/18) to 89% (16/18); corrective prompts fell from 16 to 3 across 18 runs (~5× reduction); Fisher exact p < 0.001; cost premium 9% (base), 21–29% (with tools/prompts)
• Playwright tool: Median cost +42% at High, +68% at xHigh; functional score unchanged in every matched contrast; token decomposition shows cost goes to cache re-reads (2.3M → 5.3M–7.0M median cache-read tokens), not output generation
• Docker deployment: Failed first try in 40/90 runs (44%); first-try pass rate jumped from 2/20 (Opus 4.6 High) to 17/18 (Opus 4.7 xHigh) — a shift invisible in family means differing by <1 point
• Design prompt: Full prompt → visual quality 4.5/5; abridged one-paragraph prompt → 4.7/5; both vs. 3.0 for all non-prompted runs; functional scores statistically indistinguishable from base
• Design prompt cost: Output tokens rise ~25–60% (agent writes more UI code); lines of code added rise from ~2,200 to 3,000–3,900; drag-and-drop failures elevated (11/40 prompted vs. 5/45 unprompted runs; Fisher exact p = 0.09)

Limitations

• Observational design with assigned (not randomized) conditions; p-values are descriptive summaries of association strength, not causal identification (explicitly stated per STROBE guidance)
• Single application task; generalization to other coding tasks is unverified
• Small N for Gemini (3 runs) and Qwen (2 runs); findings for those families are illustrative, not inferential
• Environment confound: agent ran in WSL/Unix-like shell; verification ran on native Windows host; platform-specific failures (native SQLite module, shell scripts) may inflate environment-criterion failure rates
• xHigh effort not available on all tested models, limiting the effort contrast to Opus 4.7
• Visual quality ratings produced by an LLM rater (though human-verified)

Relevance to Agentic AI / LLM Agents

This study is directly actionable for anyone deploying LLM coding agents: it demonstrates empirically that the dominant lever for first-try reliability is reasoning depth, not tool augmentation, and provides a concrete cost-benefit ratio (9–29% cost increase buys a ~3× lift in first-try success and ~5× reduction in human corrective interventions). The finding that browser-testing tool cost lands in cache re-reads rather than productive generation reveals a structural reason why reactive checking tools underperform on failures rooted in build and environment layers — the tool sees the interface but not the container. The criterion-level and token-level decomposition methodology (first-try rate and repair burden rather than post-repair totals; output vs. cache token attribution) offers a reusable evaluation framework for agentic benchmarking, where aggregate scores systematically compress the variance that matters for autonomous deployment.