Skip to content

StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

🕒 Published (v1): 2026-06-17 21:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

StaminaBench is a procedural benchmark that measures how many consecutive interaction turns a coding agent can sustain before failing, using REST API server construction and incremental schema modifications over up to 100 turns. Unlike fraction-of-tasks-solved metrics, it reveals that all tested models fail within 5–6 turns without feedback, and that harness choice alone produces up to a 6× performance gap for strong models.

Problem

Existing coding benchmarks (SWE-Bench, HumanEval, Commit0) evaluate single-turn or low-turn scenarios (≤25 turns), which do not reflect real "vibe-coding" sessions spanning dozens or hundreds of iterative change requests. Multi-turn evaluation is hard because ground-truth tests become inapplicable when agent and reference implementations diverge. The benchmark gap leaves harness and feedback-loop design largely unevaluated at long horizons.

Method

StaminaBench formalizes a general framework for multi-turn agent benchmarks as a coupled pair: a reference system with state space \(S\), initial distribution \(p_0\), action space \(A(s)\), transition \(\tau: S \times A \to S\), and environment policy \(\pi_{\text{env}}\); and an agent system \(\pi: \hat{S} \times D \to \hat{S}\) that tracks the reference via natural-language descriptions only.

The concrete instantiation uses REST API servers: agents receive an OpenAPI-like schema (2–3 typed entities initially), implement a server in a Docker container, then apply 100 sequentially generated change requests (entity add/delete/rename, field modifications, relationship changes, analytics endpoints—5 changes batched per turn). Tests are generated programmatically via HTTP and are fully LLM-free. After each turn, the agent may retry up to \(R\) times using configurable feedback (detailed assertion messages, test-name-only, or binary pass/fail). The primary metrics are average turns passed \(\frac{1}{K}\sum_{k=1}^{K}\sum_{t=0}^{N} \text{pass}(r_{k,t})\) and pass rate (fraction of scenarios completed end-to-end), both reported at retry budget \(@R\).

Six harnesses are evaluated: OpenCode, Mini-SWE (modified for context compaction), OpenHands, and three provider-built harnesses (Mistral Vibe, Kimi CLI, QwenCode). Seven open-weight LLMs (24B–1T parameters) are tested across 20 scenarios × 100 turns.

Key Contributions

  • A general MDP-style framework for constructing arbitrarily long multi-turn agent benchmarks with procedural generation, validity guarantees, and language-agnostic black-box testing.
  • StaminaBench: a REST API instantiation supporting 100–∞ turns, codebases growing to 6,000+ lines, reproducible seeded generation, and HTTP-based evaluation requiring no LLM in the test pipeline.
  • Empirical study across 6 harnesses × 7 models showing harness quality is a prerequisite for strong performance and that test-feedback retry loops are the single largest performance lever.

Results

  • No feedback (\(R=0\)): Best model (GLM-5) fails after just 6.2 turns on the best harness; all others fail within ~5 turns.
  • With feedback (\(R=2\)): GLM-5 + OpenCode reaches 57.0 ± 8.6 turns (25% pass rate); Qwen3.5-122B + OpenCode reaches 39.4 ± 7.1 (10% pass rate); Kimi K2.5 + provider agent reaches 33.7 ± 6.4.
  • Harness gap: Strong models show up to 6× difference between best (OpenCode) and worst (OpenHands) harness; weaker models fail with all harnesses.
  • Feedback ablation: Detailed vs. minimal feedback yields 6–12× more turns; GLM-5 drops from 57.0 to 10.7 turns without details; Qwen3.5 from 39.4 to 2.8.
  • Retry scaling: Most gains from feedback come within the first 3–5 retry attempts; performance plateaus beyond that.
  • Harness ranking: OpenCode is best or statistically indistinguishable from best for all 7 models; OpenHands is worst for 6/7; provider-built harnesses are not reliably better than OpenCode (QwenCode is significantly worse for both Qwen models, \(p < 0.02\)).
  • Language: JavaScript trends higher for some models; Rust significantly hurts Kimi K2.5, Nemotron Super, Qwen3-Coder-Next, and Qwen3.5-122B (\(p < 0.05\)).
  • Scenario generation: LLM-driven vs. programmatic change sampling produces no statistically significant performance difference for 6/7 models.

Limitations

  • Maximum 25% pass rate (GLM-5 + OpenCode) after 100 turns—no model reliably completes the full horizon, so the benchmark does not yet produce a ceiling that separates strong from perfect agents.
  • Only 20 scenarios per configuration; variance is high (SE up to ±9 turns), and some cross-model differences do not reach significance at \(n=20\).
  • All agents are instructed to use Python by default; the language ablation is limited to Python/JavaScript/Rust.
  • Failure classification relies on an LLM agent analyzing logs, introducing a secondary automated analysis step with its own reliability concerns.
  • REST API is a single domain; generalization of harness rankings to other long-horizon task types is unverified.
  • Context compaction strategies (e.g., Mini-SWE modification) are not systematically ablated—it is unclear how much compaction design drives the harness ranking.

Relevance to Harnesses / Meta-Harnesses

StaminaBench is one of the first benchmarks that treats the harness as a first-class experimental variable rather than a fixed scaffold, directly measuring how much harness design contributes to agent stamina independent of model capability. The finding that the same model achieves up to 6× more turns under OpenCode than OpenHands—and that provider-built harnesses often underperform general ones—is direct, quantified evidence that harness architecture (context management, feedback routing, retry orchestration) is a dominant factor in multi-turn performance. The paper's general framework (reference system + agent system + test function + retry loop) is itself a meta-harness template reusable across domains wherever initial state, actions, and tests can be procedurally defined. For researchers building or comparing harnesses, StaminaBench provides the first standardized long-horizon stress test with the sensitivity to distinguish harness quality effects from model effects.