Skip to content

Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality

🕒 Published (v1): 2026-07-04 03:55 UTC · Source: Arxiv · link

Why this paper was selected

Hassan group; empirical dissection of scaffolding middleware as first-class quality variable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper conducts the first longitudinal controlled study isolating the agentic scaffolding's contribution to coding agent quality by holding the LLM fixed while varying 35 sequential scaffolding releases of Qwen Code CLI. It finds no statistically significant improvement in task resolve rates across releases, while token consumption and tool calls more than double. The authors attribute this to an absence of "Agentic QA" — systematic behavioral regression testing in CI/CD.

Problem

Practitioners and researchers consistently attribute coding agent quality regressions to the underlying LLM, not the scaffolding layer. Existing benchmarking studies fix the scaffolding and vary the model, making it impossible to isolate the scaffolding's own contribution. No prior work had empirically examined whether rapid, continuous scaffolding evolution actually improves agent quality — or degrades it.

Method

The study has two parts. First, a landscape analysis mines the release history of five open-source scaffoldings (Codex, Qwen Code, Gemini CLI, OpenCode, OpenHands) versus two traditional OSS baselines (VSCode, GitHub CLI), characterizing release velocity, commit rate, PR review times, and issue accumulation. Second, a controlled longitudinal evaluation fixes a single LLM (Qwen3-Next-80B-A3B-Instruct) and evaluates every one of 35 sequential Qwen Code CLI releases against 50 stratified SWE-bench Verified tasks (two independent runs per task), computing resolve rate (effectiveness), token consumption, and tool call count (efficiency). Quality shifts are then regressed against 23 release-level development factors (code churn, commit type distribution, contributor activity, PR size) for RQ2, and commits are mapped to a reference scaffolding architecture (LLM Provider, Context Management, Tool Execution, Action Decision, Extensibility, Security) for a component-level risk analysis in RQ3. Statistical associations use Spearman \(\rho\).

Key Contributions

  • Quantifies "hyper-churn": 1.5–18 releases/week and 2.8–34 commits/day across five scaffolding projects, far exceeding mature OSS baselines, with median PR review times under four hours.
  • First controlled longitudinal evaluation (35 releases, fixed LLM) on SWE-bench Verified, isolating scaffolding impact from model variation.
  • Release-level development pattern analysis linking commit type distributions, PR size, and code deletion to quality fluctuations.
  • Component-level architectural risk map identifying LLM Provider and Context Management as high-risk zones, and Extensibility/Security as low-risk zones.
  • Comprehensive replication package including all 35 scaffolding versions, inference scripts, and raw results.

Results

  • No statistically significant improvement in resolve rate across 35 Qwen Code releases with a fixed LLM; early releases sometimes outperform later ones.
  • Token usage and tool call counts more than double in later releases without corresponding resolve rate gains.
  • Feature-heavy releases correlate positively with resolve rate (\(\rho = 0.438\)) but at the cost of increased token consumption and tool calls.
  • Fix-heavy releases associate with higher token consumption without improving resolve rates.
  • Larger average PR size correlates with lower token consumption; higher code deletion volume correlates with lower token consumption.
  • Modifications to the LLM Provider and Context Management components most frequently associate with quality degradation; Extensibility and Security changes are neutral or safe.
  • Every documented quality regression passed all existing CI/CD automated checks, confirming current testing practices cannot catch agentic behavioral regressions.

Limitations

  • Longitudinal deep dive is restricted to a single scaffolding (Qwen Code CLI) and a single LLM (Qwen3-Next-80B-A3B-Instruct), limiting generalizability across other scaffolding/model pairings.
  • Evaluation uses only 50 SWE-bench Verified tasks (a stratified sample), not the full 500, due to computational cost; this reduces statistical power for per-release comparisons.
  • SWE-bench Verified covers only Python repositories; findings may not extend to polyglot or non-software-engineering task domains.
  • Correlation-based analysis (Spearman \(\rho\)) cannot establish causal direction between development patterns and quality shifts.
  • Closed-source scaffoldings (Claude Code, GitHub Copilot) are excluded, so the hyper-churn and regression patterns may differ for proprietary systems.

Relevance to Harnesses / Meta-Harnesses

This paper is directly about the internal structure and evolution dynamics of agentic harnesses: the scaffolding is the harness, mediating between LLM reasoning and tool execution. The finding that scaffolding changes — not model changes — drive quality regressions is a fundamental empirical result for anyone building or evaluating meta-harnesses, because it implies that harness versioning and behavioral regression testing must be treated as first-class engineering concerns. The architectural risk map (LLM Provider and Context Management as high-risk zones) gives concrete guidance on where a harness developer should concentrate QA effort. The "Agentic QA" recommendation — automated benchmark-level behavioral testing in CI/CD — is precisely the kind of meta-harness infrastructure gap this line of work must address.