Skip to content

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

🕒 Published (v1): 2025-12-21 00:49 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Self-play SWE-RL (SSR) trains a single LLM to simultaneously generate and repair software bugs using only raw sandboxed codebases—no human-labeled issues, test suites, or language-specific infrastructure. The agent plays dual roles (bug injector and bug solver) and jointly optimizes both via RL, achieving consistent self-improvement and outperforming human-data baselines on SWE-bench Verified (+10.4 pts) and SWE-Bench Pro (+7.8 pts).

Problem

Existing software engineering agents rely on human-curated training data (issue descriptions, test cases, test runners), which creates a fundamental scalability ceiling: the agents learn to replay human development traces rather than autonomously discover new problem classes. Static synthetic bug generators (e.g., SWE-smith, BugPilot) still depend on teacher models, existing test infrastructure, or fixed pipelines that cannot adapt to the evolving agent policy.

Method

SSR instantiates a single shared-weight LLM policy in two roles via different prompting:

Bug-injection agent: Given only a Docker image (source + deps), the agent explores the repo, discovers how to run tests, and produces a bug artifact consisting of five components: bug_inject.diff (code patch introducing the bug), test_script.sh (test runner), test_files.txt (oracle test list), test_parser.py (JSON test-result parser), and test_weaken.diff (patch that hides the bug from existing tests). Artifacts pass a multi-step consistency validation (test parser validity, bug validity, test-weakening validity, inverse mutation testing).

Two injection strategies improve over naive prompting: (1) removal-only—removing code hunks/files while ensuring runnability; (2) removal + history—randomly mixing removal with reversion of historical git commits.

Bug-solving agent: Receives the buggy codebase plus the reversed test-weakening patch as a formal specification (not natural language). Produces a repair patch validated against the restored oracle tests.

Higher-order bugs: Failed solver attempts become new bug artifacts, extending the training distribution to multi-step, layered failures.

Reward design: Injector reward \(r_{\text{inject}}\) penalizes artifacts failing consistency checks (−1.0), penalizes degenerate solve rates \(s=0\) or \(s=1\) (−α, α=0.8), and rewards intermediate difficulty via \(1-(1+\alpha)s\). Solver reward is binary: +1 if all tests pass, −1 otherwise. Both signals jointly update the shared policy.

Key Contributions

  • First self-play RL framework for software engineering that requires only raw sandboxed Docker images—no issue descriptions, test suites, or test commands.
  • Formal bug artifact specification enabling executable, verifiable training signals without natural language issue descriptions.
  • Higher-order bug construction from solver failures, creating an auto-curriculum that tracks the agent's evolving policy.
  • Inverse mutation testing as a consistency check ensuring each modified file is causally necessary for the bug.
  • Empirical demonstration that self-generated tasks yield richer learning signals than human-curated data throughout the entire training trajectory.

Results

  • SWE-bench Verified: SSR achieves +10.4 points of self-improvement over the base model (CWM-sft, 32B) and consistently outperforms the human-data baseline (standard agentic SWE-RL with issue descriptions + test suites) across all training steps.
  • SWE-Bench Pro: SSR achieves +7.8 points self-improvement over base model and likewise dominates the human-data baseline throughout training.
  • Self-play vs. ablations (SWE-bench Verified + Pro averaged, 1231 tasks): Self-play outperforms injection-only (~28.5% vs. lower) and repair-only (~31.1% vs. lower) variants; exact numbers visible in Figure 9a as ~33–34% for full SSR vs. ~29% repair-only and ~27% injection-only at 150 steps.
  • Bug injection strategy: Removal + history > removal-only > direct-injection on both benchmarks (Figure 9b).
  • Solver feedback in reward: Marginal benefit; consistency-check-only reward achieves near-identical performance (Figure 9c).
  • Generalization: improvements transfer to natural language issue descriptions not seen during SSR training.

Limitations

  • Requires sandboxed Docker images with dependencies pre-installed; cannot yet operate on truly arbitrary codebases without some environment setup.
  • Evaluation uses only one attempt per problem with no parallel test-time scaling, potentially underestimating achievable performance.
  • Solver feedback signal is weak and noisy; the reward's solve-rate term provides negligible advantage over a simpler consistency binary reward.
  • Direct-injection strategy collapses to trivial one-line bugs; strong performance depends on carefully designed injection prompts.
  • Higher-order bugs are capped at second order to avoid overlap with existing bugs, which may limit curriculum depth.
  • Base model (CWM-sft) and infrastructure are from Meta FAIR, creating a potential conflict of interest acknowledged by the authors.
  • The bug-injection capability could be misused to deliberately introduce vulnerabilities or misleading code artifacts.

Relevance to Agentic AI / LLM Agents

SSR is a direct advance in autonomous agent self-improvement: it demonstrates that a single LLM agent can bootstrap its own training curriculum from raw environment interaction without human supervision, analogous to AlphaZero's rule-only self-play but grounded in real-world software repositories. The dual-role (proposer/solver) architecture and formal verifiable reward signal are a generalizable template for agentic RL in domains with executable environments. The result that self-generated tasks outperform human-curated data challenges the assumption that human annotations are a necessary ceiling, with implications for any agentic domain where execution-based verification is feasible. The work also surfaces an important design principle: agent-adaptive, online task generation (self-play) is strictly superior to static synthetic data pipelines for sustained capability improvement.