DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks¶

🕒 Published (v1): 2026-07-08 21:45 UTC · Source: Arxiv · link

Why this paper was selected

DeepSWE: original long-horizon coding benchmark without data-leakage from GitHub mining

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DeepSWE is a 113-task benchmark for evaluating coding agents on original, long-horizon software engineering tasks, designed to eliminate benchmark contamination and poor verifier fidelity that afflict SWE-bench-derived benchmarks. Tasks are written from scratch across 91 repositories and five languages, graded by hand-written functional verifiers. On the leaderboard, gpt-5.5 leads at 70% pass@1, while an independent LLM-judge audit finds DeepSWE verifiers disagree with its judge only 1.4% of the time versus 32.4% for SWE-Bench Pro.

Problem¶

SWE-bench-lineage benchmarks suffer two structural flaws: (1) contamination — fixes and discussions are public and likely in pretraining corpora, so high scores can reflect recall rather than problem-solving; (2) verifier infidelity — inherited pull-request tests were written to confirm one specific fix and can reject valid alternative implementations or pass incomplete ones. An independent LLM-judge audit finds SWE-Bench Pro's verifiers disagree with its judge on ~32% of audited rollouts.

Method¶

Task authoring: 113 tasks written from scratch across 91 active open-source repositories (TypeScript 31%, Go 30%, Python 30%, Rust/JS 5% each); tasks are never merged upstream, keeping reference solutions off the public commit record.
Verifier design: Each task ships a hand-written functional verifier extending the repository's own test infrastructure. Verifiers assert through public APIs and observable outputs only, accepting any correct implementation rather than one specific shape.
Harness: All 16 model configurations run through a fixed, model-agnostic mini-swe-agent with a single bash tool and shared prompt; no per-vendor editing primitives. Wall-clock timeout of 9,000 s; no step or cost cap.
Quality assurance: Each verifier runs 3× during authoring; flaky verifiers are revised. Regression tests guard against unrelated breakage. LLM-assisted analysis plus independent human review over prompt, verifier, reference solution, and diagnostic rollouts before inclusion.
Verifier audit: 30 tasks drawn from DeepSWE and SWE-Bench Pro each, 3 rollouts across 9 frontier configurations; GPT-5.5 (xhigh) run as a Codex CLI agent independently re-grades every trajectory.

Key Contributions¶

Original benchmark of 113 tasks spanning 91 repositories and 5 languages, with no reference solutions in the public commit record.
Functional verifiers purpose-written from task descriptions; disagreement rate with an independent LLM judge is 1.4% vs. 32.4% for SWE-Bench Pro.
Substantially harder tasks from shorter prompts: DeepSWE prompts average 2,158 characters versus 4,614 for SWE-Bench Pro, yet reference solutions average 668 lines added and 7.4 files edited versus 120 lines and 5.1 files.
Standardized harness isolates model capability from scaffold choice; a pilot on 10 SWE-Bench Pro tasks found mini-swe-agent pass rates equal or superior to native harnesses for all three tested models.
Full release of benchmark, verifiers, and evaluation trajectories.

Results¶

gpt-5.5 [xhigh]: 70.0% pass@1 (88% pass@4)
gpt-5.4 [xhigh]: 55.5% pass@1 (77% pass@4)
claude-opus-4.7 [max]: 54.2% pass@1 (86% pass@4)
claude-sonnet-4.6 [high]: 31.6% pass@1 (62% pass@4)
gemini-3.5-flash [medium]: 28.3% pass@1 (50% pass@4)
claude-opus-4.6 [max]: 27.1% pass@1 (46% pass@4)
gpt-5.4-mini [xhigh]: 24.3% pass@1 (49% pass@4)
kimi-k2.6: 23.9% pass@1 (45% pass@4)
deepseek-v4-pro: 17.5% pass@1 (39% pass@4)
mimo-v2.5-pro: 19.5% pass@1 (57% pass@4)
gemini-3.1-pro: 9.9% pass@1 (25% pass@4)
glm-5.1: 7.5% pass@1 (19% pass@4)
gemini-3-flash: 5.2% pass@1 (15% pass@4)
qwen3.6-plus: 2.7% pass@1 (10% pass@4)
claude-haiku-4.5 / minimax-m2.7: 0.2% pass@1
Verifier audit: SWE-Bench Pro false-positive rate 8.5%, false-negative rate 24.0%; DeepSWE 0.3% and 1.1% respectively.
Only 67 of 7,174 scored rollouts (0.9%) hit the 9,000 s timeout.

Limitations¶

113 tasks is a relatively small corpus; individual DeepSWE verifier error rates rest on single-digit raw counts.
Reasoning-effort settings were not swept consistently across model families; some models ran at provider defaults rather than explicitly tuned effort levels, limiting comparability.
Standardized mini-swe-agent harness may not reflect real-world performance through native products (Codex CLI, Claude Code, Cursor); the pilot checking this used only 10 tasks per model.
Tasks skew toward long-horizon multi-file implementation; short single-file edits and bug localization are under-represented, so the benchmark does not measure the full spectrum of coding agent capability.
The LLM-judge audit is a disagreement rate between two independent readers, not a ground-truth error rate; the judge itself can be wrong.
One contamination leakage vector was found in SWE-Bench Pro containers (.git history exposing reference fixes); DeepSWE mitigates this with shallow clones, but no contamination study is reported on DeepSWE itself.

Relevance to Agentic AI / LLM Agents¶

DeepSWE directly addresses the benchmark quality problem that limits trustworthy measurement of autonomous coding agents: contamination inflating scores and verifier noise obscuring capability differences. Its design — short, underspecified prompts requiring end-to-end repository exploration — more faithfully models agentic workflows where the agent must itself discover what to change and where. The wide score separation it induces across frontier models (0.2% to 70%) provides a more discriminative signal for tracking progress than current leaderboards where models cluster. For researchers studying agent scaffolds, the standardized harness and published trajectories also enable controlled analysis of failure modes across model families.