Skip to content

Kimi-Dev: Agentless Training as Skill Prior for SWE-agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR Kimi/Moonshot; agentless training as skill prior for SWE-agent harness design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Kimi-Dev reframes agentless (workflow-based) SWE training not as an end in itself but as a source of skill priors—localization, code edit, self-reflection—that enable efficient adaptation into multi-turn SWE-agents. Training a 72B model with a four-stage agentless recipe (mid-training → cold-start SFT → RLVR → test-time self-play) achieves 60.4% on SWE-bench Verified (SOTA among workflow approaches). A subsequent 5k-trajectory SFT lifts the same model to 48.6% pass@1 under an agentic framework, matching Claude 3.5 Sonnet (241022).

Problem

Direct SWE-agent training suffers from sparse outcome rewards over long horizons, brittle initialization (e.g., tool-use failures, infinite action loops), and expensive multi-turn trajectory collection. The field has treated agentless pipelines (fixed, modular, single-turn) and SWE-agents (end-to-end, multi-turn) as mutually exclusive paradigms, leaving no principled method to transfer structured workflow knowledge into agentic models.

Method

Phase 1 – Agentless recipe (Kimi-Dev, 72B): - Mid-training on ~150B tokens from Qwen 2.5-72B-Base: ~50B agentless-derived natural diffs, ~20B PR commit packs, ~20B synthetic reasoning/agentic data (4× upsampled); strict decontamination of SWE-bench Verified repositories. - Cold-start SFT using DeepSeek R1-generated BugFixer/TestWriter reasoning trajectories from SWE-Gym and SWE-bench-extra, activating long CoT (problem analysis, method sketching, self-refinement). - RLVR on the code-edit stage only (localization already strong post-SFT). Binary outcome reward: BugFixer patch passes all ground-truth unit tests; TestWriter test fails on unfixed repo and passes on fixed repo. Adaptive curriculum: start with 1,200 problems (pass@16 > 0), inject 500 harder problems every 100 RL steps once success rate plateaus; late-stage positive example reinforcement from recent iterations. Infrastructure: Kubernetes-managed Docker sandboxes supporting >10,000 concurrent rollout instances. - Test-time self-play: generate \(|B|=40\) BugFixer patches and \(|T|=40\) TestWriter tests per instance. Invalid tests (not failing on unfixed repo) are discarded. Score each patch \(b_i\) with $\(S_i = \sum_j \frac{\text{FP}(i,j)}{F(j)} + \sum_j \frac{\text{PP}(i,j)}{P(j)}\)$ aggregating fail-to-pass (reproduction) and pass-to-pass (regression) signals across all test pairs; select \(\arg\max_i S_i\).

Phase 2 – SWE-Agent adaptation: - Minimal SFT on 5,016 publicly available SWE-smith trajectories (collected with Claude 3.7 Sonnet); no additional multi-turn agentic RL. Training context 64K tokens; inference up to 128K tokens / 100 turns under SWE-Agent's five-stage system prompt.

Key Contributions

  • Empirical demonstration that agentless and agentic frameworks are complementary: agentless training induces transferable atomic skill priors.
  • Open-source Kimi-Dev-72B achieving 60.4% on SWE-bench Verified—SOTA among workflow-based solutions.
  • Lightweight SFT from Kimi-Dev priors reaches 48.6% pass@1 in agentic mode, matching Claude 3.5 Sonnet (241022) at 49%.
  • Cross-execution self-play scoring (Eq. 1) using BugFixer Ă— TestWriter matrices; 3Ă—3 self-play already outperforms 40-patch majority voting.
  • Ablation establishing each training stage's marginal contribution to downstream SWE-agent adaptation efficiency (data-efficiency curves across 100–5,016 trajectory subsets).
  • Scalable RL infrastructure: >10,000 concurrent Kubernetes Docker instances for outcome-based SWE reward.

Results

  • Agentless framework (workflow), Kimi-Dev 72B: 60.4% on SWE-bench Verified vs. SWE-SWISS 58.2%, DeepSeek-R1-0528 57.6%, MiniMax-M1 56.0%, Claude 3.5 Sonnet (241022) 50.8%.
  • Agentic framework (SFT'd), Kimi-Dev 72B: 48.6% pass@1, vs. SWE-agent-LM 42.2% (same SFT data), Devstral-Small-2507 53.6%, gpt-oss-20b 60.7% (20B, internal system).
  • Agentic pass@10: 74.0%, exceeding agentless pass@30 (73.8%), confirming higher ceiling.
  • Self-play (40Ă—40): improves from 48.0% (1Ă—1) to 60.4%; consistently surpasses BugFixer majority voting at all tested pair counts.
  • Mid-training: monotonic performance increase from 50B → 100B → ~150B tokens.
  • RL: pass rate and response length increase jointly and consistently throughout training.

Limitations

  • Self-play scores remain below pass@N with ground-truth test oracle, indicating TestWriter quality remains a bottleneck.
  • SWE-Agent adaptation relies on 5k synthetic-environment trajectories; no additional multi-turn agentic RL was performed.
  • TestWriter encounters false-positive examples during RL due to insufficient reproduction test coverage.
  • RLVR is applied only to the code-edit stage; localization is not jointly RL-trained.
  • Generalization beyond SWE-bench Verified (SWE-bench-live, SWE-bench Multilingual) is mentioned but not deeply analyzed in the main text.

Relevance to Harnesses / Meta-Harnesses

The entire Kimi-Dev training and evaluation pipeline is mediated by an auto-eval harness (unit-test execution as oracle reward), making harness design the central axis of both training signal validity and benchmark soundness. More directly, the BugFixer+TestWriter duo constitutes a two-component meta-harness where one model generates evaluation artifacts (tests) that serve as the scoring harness for another model's solution artifacts (patches)—a learned, inline test-generation harness nested inside the fix-generation pipeline. The self-play scoring function (Eq. 1) is itself a programmatic meta-harness that aggregates cross-execution signals over a patch×test matrix rather than relying on fixed human-curated test suites. This architecture—where agent-generated components serve double duty as both solutions and evaluation harnesses—is a concrete instantiation of the broader meta-harness design principle: decouple solution generation from evaluation, then let both components co-evolve.