Skip to content

Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?

🕒 Published (v1): 2025-11-17 17:58 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Live-SWE-Agent is a runtime self-evolving software engineering agent that starts from a minimal bash-only scaffold (~100 lines) and autonomously synthesizes custom tools on the fly during issue resolution, requiring no offline training. It achieves 77.4% on SWE-bench Verified and 45.8% on SWE-Bench Pro (both state-of-the-art at time of writing), outperforming much heavier hand-crafted scaffolds and prior self-improving agents.

Problem

Existing software agents use fixed scaffold designs whose action spaces are preset regardless of the task, making them suboptimal. Prior self-evolving agents (DGM, HGM, SICA) require massive offline training (500–1231 GPU-hours, ~$22,000 per DGM run) on specific benchmarks, producing static agents that fail to generalize across tasks and LLMs. The fundamental challenge is that exhaustive manual scaffold design is intractable given the infinite design space.

Method

Live-SWE-Agent adds two minimal modifications to any base agentic loop (instantiated on mini-SWE-agent, ~100 LOC, bash-only):

  1. Initial prompt injection: Instructions and examples telling the agent it may create executable script files as custom tools at any time, for any purpose, including task-specific ones.
  2. Step-reflection prompt: After every environment feedback message, a lightweight reflection message asks the agent whether creating or revising a tool would accelerate progress.

Custom tools are plain scripts dropped into the environment's tools/ directory and invoked like any shell command. The agent iteratively synthesizes, refines, and reuses these tools throughout the issue-solving trajectory. No changes to the agentic loop, no offline training, no workflow constraints. The mechanism is LLM-agnostic and scaffold-agnostic.

Key Contributions

  • First live, runtime self-evolving software agent: on-the-fly tool synthesis without offline training cost.
  • Minimal implementation: two prompt additions to any base scaffold; fully open-source.
  • State-of-the-art results: 77.4% SWE-bench Verified (no test-time scaling), 45.8% SWE-Bench Pro.
  • Outperforms offline self-improving agents (DGM 53.3% → Live-SWE-Agent 65.0% on Verified-60 subset) with zero offline cost vs. 1231 hours for DGM.
  • Unified open leaderboard enabling fair apples-to-apples model comparison via a shared scaffold.

Results

  • SWE-bench Verified: 77.4% resolve rate with Gemini 3 Pro; 75.4% with Claude 4.5 Sonnet. Baseline mini-SWE-agent: 74.2% / 70.6% respectively. Consistent +2–5 pp gain across all four tested LLM backends.
  • SWE-Bench Pro: 45.8% vs. SWE-agent 43.6% (both Claude 4.5 Sonnet). Best-known result at time of writing.
  • Verified-60 subset (vs. offline self-improvers, all GPT-5-Mini): Live-SWE-Agent 65.0% vs. HGM 56.7%, DGM 53.3%, SICA 50.0%; offline cost 0 hours vs. 512–1231 hours.
  • Cost overhead: marginal — e.g., Claude 4.5 Sonnet $0.56 (mini) → \(0.68 (Live); GPT-5 actually shows slight savings (\)0.28 → $0.27).

Limitations

  • Current implementation focuses solely on tool creation; other scaffold components (planner, memory, verifier) are not evolved.
  • Tool synthesis quality is bounded by underlying LLM capability; weaker models benefit less.
  • Evaluated only on Python-heavy SWE-bench and multi-language SWE-Bench Pro — generalization to non-SE agentic tasks is unvalidated.
  • No formal analysis of tool quality or correctness; erroneous synthesized tools could mislead the agent.
  • Single-attempt evaluation; interaction with test-time scaling methods not studied.
  • The reflection prompt is simple string heuristics — no principled trigger for tool creation.

Relevance to Harnesses / Meta-Harnesses

Live-SWE-Agent is a direct instantiation of a runtime meta-harness: the scaffold itself is the target of modification, and the agent acts as its own harness engineer by synthesizing, registering, and iterating on tools mid-execution. This is architecturally distinct from static meta-harnesses (e.g., DGM) that bake improvements offline before deployment — Live-SWE-Agent demonstrates that a harness can grow its own action space dynamically within a single run. For researchers building meta-harnesses, the paper provides concrete evidence that a two-prompt modification is sufficient to unlock runtime scaffold extension, and that this beats hundreds of hours of offline scaffold engineering. The unified leaderboard contribution also points toward a shared-harness evaluation norm, relevant to anyone designing benchmark infrastructure for agent scaffolds.