Skip to content

SWEnergy: An Empirical Study on Energy Efficiency in Agentic Issue Resolution Frameworks with SLMs

🕒 Published (v1): 2025-12-10 11:28 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWEnergy is a controlled empirical study measuring energy consumption of four agentic issue-resolution frameworks (SWE-Agent, OpenHands, Mini SWE Agent, AutoCodeRover) when constrained to small language models (Gemma-3 4B, Qwen-3 1.7B) on local hardware. Across 1,200 runs on SWE-bench Verified Mini, task resolution rates were near-zero while energy waste was substantial, revealing that framework architecture—not model size alone—is the primary driver of consumption. The study concludes that existing frameworks, designed assuming capable LLMs, fail systematically with SLMs.

Problem

Existing agentic software engineering frameworks (SWE-Agent, OpenHands, AutoCodeRover) are designed for large, proprietary LLMs and their energy footprint in resource-constrained, local-deployment settings using SLMs is unmeasured. It is unknown whether SLMs can substitute for LLMs in these frameworks, and if not, where efficiency losses occur and why.

Method

Controlled experiment on fixed hardware (Intel Xeon w3-2435, 32 GB RAM, NVIDIA RTX A2000 16 GB) running four frameworks Ă— two SLMs Ă— 50 SWE-bench Verified Mini tasks Ă— 3 repetitions = 1,200 runs. Energy measured via Intel RAPL (CPU) and NVIDIA NVML (GPU) with idle baseline subtracted (51.69 W CPU, 2.70 W GPU). A custom monitoring script ran concurrently with each agent. Failure modes classified using the Multi-Agent System Failure Taxonomy (MAST). Qualitative log analysis was assisted by Gemini 2.5 Flash, with manual author spot-checks for validity.

Key Contributions

  • First end-to-end hardware-level energy analysis of multi-turn agentic workflows powered by SLMs.
  • Reproducible methodology (RAPL + NVML, idle-subtracted) with full replication package (code, logs, metrics).
  • Empirical characterization of the performance-efficiency frontier: framework architecture, not model size, is the primary energy driver.
  • Taxonomy of failure signatures per framework architecture class (ReAct-style vs. deliberative pipeline vs. general-purpose).
  • Actionable design guidance: adaptive loop-breaking, guided context filtering, and independent verification layers as required framework primitives for SLM compatibility.

Results

  • Resolution rates: AutoCodeRover (Gemma) 0%, AutoCodeRover (Qwen) 4% (2/50 tasks); all other configurations 0%.
  • Energy spread (9.4Ă— range): AutoCodeRover mean ~216 kJ (Gemma) / ~208 kJ (Qwen); OpenHands mean ~23 kJ (Gemma) / ~23 kJ (Qwen).
  • Energy correlates: Total energy strongly correlated with wall-clock duration (R=0.89) and output tokens (R=0.88); memory shows near-zero correlation.
  • Failure signatures: ReAct-style agents dominated by Step Repetition → Context Loss; AutoCodeRover by Follow Task Requirements failures and Timeout as secondary consequence; OpenHands broad-spectrum failure.
  • False positives in Mini SWE Agent: Low energy masked premature terminations and destructive patches (e.g., replacing a 2,000-line file with a single invalid line) reported as "success."
  • Runtimes: AutoCodeRover configurations mean 24–27 minutes per run.

Limitations

  • Only two SLMs tested (≤4B parameters); results may not generalize to mid-size models (7B–70B) or LLM-level models.
  • Single hardware platform (Intel/NVIDIA); AMD or Apple Silicon energy profiles may differ.
  • 50-task SWE-bench Verified Mini subset; may not represent full distribution of real-world issues.
  • Energy measurement excludes RAM, storage, and motherboard components (only CPU+GPU via RAPL/NVML).
  • Near-zero success rate means conclusions about energy cost of successful resolutions are statistically unreliable.
  • Qualitative log analysis used Gemini 2.5 Flash as an intermediary, introducing potential interpretation bias.

Relevance to Harnesses / Meta-Harnesses

This paper directly characterizes four prominent agentic harnesses (SWE-Agent, OpenHands, AutoCodeRover, Mini SWE Agent) as execution containers with distinct architectural properties, and shows that harness design choices—loop control, context management, tool orchestration strategy—dominate resource consumption independently of the underlying model. For someone tracking harness/meta-harness design, the key finding is that passive orchestration architectures (which assume a competent backbone model) fail catastrophically when that assumption is violated, pointing to the need for harnesses with active model-awareness: loop-detection, adaptive strategy switching, and external verification layers baked into the orchestration layer rather than delegated to the model. The failure taxonomy and energy measurement methodology also provide a concrete evaluation protocol applicable to any multi-agent harness.