Skip to content

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

🕒 Published (v1): 2026-06-29 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWE-INTERACT converts single-turn SWE benchmarks into multi-turn, user-driven coding sessions by pairing a coding agent with a persona-conditioned user simulator that starts with vague instructions, progressively reveals requirements, and inspects the agent's workspace using tools. Across five frontier models, resolve rates drop by roughly half (from ~25–50% to ~15–27%) when switching from the single-turn autonomous setting to the interactive setting. The benchmark exposes a distinct capability axis — interactive goal discovery and iterative refinement — not captured by existing SWE evaluations.

Problem

Existing SWE benchmarks hand the agent fully-specified requirements upfront and evaluate autonomous implementation. Real developer-agent workflows are fundamentally different: users send vague or incomplete initial instructions, critique implementations incrementally, and reveal constraints only after reviewing the agent's work. Prior multi-turn benchmarks either provide well-specified starting prompts (narrowing the gap), use static LLM calls that cannot inspect the workspace, or impose predetermined (non-interactive) turn sequences. No benchmark systematically measures whether an agent can discover user intent, incorporate evolving requirements, and build on its own prior work over a realistic interactive trajectory.

Method

Dataset. 75 tasks drawn from three frontier SWE benchmarks (SWE-bench Pro, SWE Atlas Refactoring, DeepSWE), 25 each, selected for layered requirements that naturally decompose into multi-turn interactions.

User Simulator. A separate containerized LLM agent (distinct from the solver) that holds the full task goal as latent state. It is conditioned on an Expert Nitpicker persona derived from analysis of thousands of real sessions in the SWE-chat dataset (senior developers who are brief, direct, iterative, and demand exact API compliance). The simulator starts with a high-level vague description and reveals sub-goals one at a time only after inspecting the solver's workspace via shell tools (git, grep, sed, find). This makes the user dynamic and agentic rather than a static API call.

Execution. Tasks run in Harbor (containerized Modal sandboxes). The solver agent has access to an MCP-hosted ask_user tool. It commits a plan (PLAN.md) and each implementation revision triggered by user feedback; the original verifier (unit tests + rubrics) scores the final submission.

Goal Discovery Analysis. Sub-goals are extracted from full task instructions by GPT 5.5. A Sonnet 4.6 high judge scores the agent's plan and each committed implementation checkpoint against these sub-goals, yielding a lifecycle trace of goal coverage vs. trajectory progress.

Failure Mode Analysis. 287 failed trajectories are labeled by GPT 5.5 against five semantic categories: Technical implementation bug, Forgotten requirement, Misinterpretation/Bad assumption, Missing user requirement, Regression.

Ablations. (1) Persona: Expert Nitpicker vs. neutral generic prompt. (2) Simulator model: Opus 4.7 vs. GPT 5.5.

Key Contributions

  • A 75-task testbed adapting real SWE benchmarks into interactive multi-turn coding sessions, with Docker environments and original verifiers preserved.
  • A persona-conditioned, workspace-aware user simulator derived from large-scale real coding-session data (SWE-chat), capable of running shell commands in the agent's workspace.
  • Quantitative evidence that strong single-turn SWE performance does not predict multi-turn performance: best models drop ~23–24 pp in resolve rate.
  • A goal-discovery lifecycle analysis showing that frontier models can recover >90% of latent goals via iteration, yet still fail on correctness.
  • A failure-mode taxonomy (technical bugs 34%, forgotten requirements 34%, misinterpretation 14%, missing requirements 12%, regressions 7%) characterizing why multi-turn settings are harder.
  • Ablation studies showing that (a) Expert Nitpicker persona increases interaction length by ~30–50% and reduces resolve rates vs. neutral baseline, and (b) GPT 5.5 as simulator is qualitatively superior to Opus 4.7, using 4–5× more workspace tool calls.

Results

  • Resolve rate drop: All five models score lower in multi-turn vs. single-turn. GPT 5.5: 48.0% → 24.7% (−23.3 pp); Opus 4.8: 50.7% → 26.7% (−24.0 pp); Kimi K2.6: 25.3% → 14.7% (−10.7 pp); Gemini 3.5 Flash: 29.3% → 17.3% (−12.0 pp); Sonnet 4.6: 21.3% → 18.8% (−2.5 pp).
  • Compute overhead: Multi-turn trajectories use 3.2–3.9× more steps and 1.6–4.5× more tokens per trial; cost per trial increases 2.1–3.5×.
  • User interaction volume: Average ~7–11 user messages per trial; up to 27 user messages and 332 user tool calls in the longest trajectory.
  • Goal discovery: GPT 5.5, Opus 4.8, and Sonnet 4.6 eventually address >90% of task goals on average; Kimi K2.6 falls significantly short.
  • Goal vs. correctness: Near-perfect goal-discovery score is necessary but not sufficient for verifier success; many agents discover all goals yet still fail correctness checks.
  • Persona ablation: Expert Nitpicker persona reduces resolve rate for 4/5 models vs. neutral baseline while increasing agent steps by 29–49% and user-agent turns by ~90–97%.
  • Simulator model ablation: GPT 5.5 simulator causes 1.5–2× longer agent trajectories and 4–5× more user tool calls compared to Opus 4.7; GPT 5.5 simulator improves final resolve rate for Opus 4.8 (22.7% → 30.7%) and Gemini (14.7% → 20.0%).

Limitations

  • Only 75 tasks total (25 per benchmark); single-turn results run once (no confidence intervals) due to cost.
  • User simulator is limited to read-only shell commands (git, grep, sed, find); it cannot modify code, run tests, or take write actions, which real users often do.
  • A single Expert Nitpicker / vibecoding persona is modeled; no coverage of other common user types (beginners, verbose users, non-technical stakeholders).
  • ~12% of failure labels are "Missing user requirement," flagging simulator robustness gaps where the simulator failed to surface a needed requirement — benchmark false negatives.
  • User simulator is verifier-blind; it cannot verify functional correctness, only inspect code artifacts.
  • Multi-turn results are averaged over only two simulator model runs (Opus 4.7 and GPT 5.5), limiting statistical power despite Wilson 95% CI reporting.

Relevance to Agentic AI / LLM Agents

SWE-INTERACT directly challenges a core assumption in agentic benchmarking — that autonomous task completion under full specification is a reliable proxy for real-world utility. By demonstrating a consistent ~50% relative drop in resolve rates when requirements are revealed interactively, it quantifies a capability gap in goal discovery and iterative refinement that current agent training and evaluation largely ignores. The persona-conditioned, workspace-aware user simulator design offers a concrete methodology for building more realistic interactive evaluation harnesses, extending the tau-bench line of work into the SWE domain. For researchers building or evaluating LLM agents, the failure taxonomy (especially the dominance of "forgotten requirements" and "technical bugs" over "failure to elicit information") suggests that memory across turns and correctness under revision pressure are more critical bottlenecks than clarification-seeking behavior alone.