Skip to content

SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models

🕒 Published (v1): 2025-11-07 18:01 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWE-Compass is a 2,000-instance benchmark for evaluating LLM agentic coding abilities across 8 task types, 8 programming scenarios, and 10 programming languages, sourced from real GitHub pull requests. It addresses the narrow fixation on Python bug-fixing that dominates existing SWE benchmarks by providing a structured, execution-grounded evaluation framework. Ten state-of-the-art LLMs are evaluated under two agentic frameworks (SWE-Agent and Claude Code), exposing systematic capability gaps.

Problem

Existing coding benchmarks are dominated by Python-centric bug fixing (SWE-bench and variants) or isolated algorithmic problems (HumanEval, MBPP), leaving critical developer activities — feature implementation, refactoring, configuration/deployment, performance optimization, test generation — systematically unmeasured. This narrowness prevents fine-grained capability diagnosis and makes it impossible to distinguish general reasoning from artifact-specific overfitting.

Method

Benchmark construction proceeds through five stages:

  1. User Analysis via Active Learning: Real-world coding discussions from GitHub and Stack Overflow are iteratively labeled by Qwen3-Coder-30B-A3B-Instruct using in-context learning; tag clustering and LLM-guided seed optimization iterate to convergence (5 rounds), yielding a stable taxonomy of 8 task types, 8 scenarios, and 10 languages.

  2. Data Collection: ~50,000 high-quality PRs filtered from repositories with \(\geq\)500 stars, active maintenance, \(\geq\)3 contributors, \(\geq\)1000 issues/PRs. PRs must be merged, linked to Issues, and carry complete metadata (commit, test patch, code patch).

  3. Environment Building: Dockerfiles are auto-generated from dependency manifests; expert annotators (30) repair build failures. Initial automated build success rate ≈2%; after expert retry on Kubernetes, retention reaches ≈8%, yielding ~4,000 runnable Docker images validated via F2P/P2P consistency checks.

  4. Task Construction via three complementary strategies:

  5. Checklist Synthesis (Code Understanding): GPT-5 generates natural-language queries + reasoning checklists for LLM-as-a-Judge evaluation.
  6. Reverse Masking (Configuration & Deployment, Test Case Generation): controlled perturbations to "golden" artifacts to create reproducible failure cases.

  7. Heuristic Filtering (Bug Fixing, Feature Implementation/Enhancement, Refactoring, Performance Optimization): PR classification by patch intent and behavioral context; Performance Optimization requires \(>30\%\) runtime improvement.

  8. Data Validation: difficulty filtering by modified-file count, changed-line count, and multi-model inference signals; balanced sampling; manual expert verification.

Evaluation metrics are task-specific: Pass@1 for patch-based tasks; Performance Optimization Score (binary: runtime \(< 80\%\) of baseline); Line Coverage for test generation; LLM-as-a-Judge Score (checklist pass rate) for Code Understanding.

Key Contributions

  • A 2,000-instance execution-grounded benchmark spanning 8 task types, 8 programming scenarios, and 10 languages, sourced from real GitHub PRs with reproducible Docker environments.
  • An iterative Active Learning pipeline for taxonomy discovery that grounds label space in empirical user-need distributions rather than researcher intuition.
  • A systematic evaluation of 10 state-of-the-art LLMs (including Claude Sonnet 4, GPT-4.1, Gemini 2.5 Pro, Qwen3-Coder-480B, Kimi-K2) under two agentic frameworks (SWE-Agent and Claude Code).
  • Task-specific evaluation protocols combining Pass@1, execution timing, line coverage, and LLM-as-a-Judge to handle heterogeneous output types.

Results

  • Task-type resolve rates reveal a clear difficulty hierarchy: Bug Fixing achieves the highest resolve rates (Claude Sonnet 4: 28.2%), while Configuration & Deployment, Performance Optimization, and Test Case Generation are substantially harder.
  • Claude Sonnet 4 leads overall, followed by Qwen3-Coder-480B, Kimi-K2-Instruct, Gemini 2.5 Pro, and GPT-4.1 in the task-specific resolve rate figures shown.
  • Feature Implementation (42.1%–56.8% top-model range visible in figure) and Feature Enhancement perform better than Performance Optimization or Code Understanding.
  • SWE-Compass is the only benchmark in the comparison table with 10 languages, 8 task types, and 2,000 samples at an average of 4.7 modified files per instance — closest competitor (SWE-bench-Pro) has 1,865 samples, 4 languages, 4 task types.

(Note: full numeric tables are truncated in the provided text; figures cited above are read from Figure 1.)

Limitations

  • Initial Docker environment build success rate is only ~2% (reaching ~8% with expert intervention), meaning the benchmark is biased toward repositories whose dependency graphs are resolvable — potentially underrepresenting cutting-edge or less-maintained ecosystems.
  • Code Understanding tasks rely on LLM-as-a-Judge evaluation (GPT-5-generated checklists), introducing judge model bias and reproducibility concerns.
  • Performance Optimization instances require \(>30\%\) runtime improvement threshold, which may not capture meaningful but smaller gains.
  • The benchmark uses a fixed budget/resource cap per evaluation; results may not generalize to unconstrained agent settings.
  • Synthesis (reverse masking, checklist generation via GPT-5) introduces distributional shift relative to purely organic task instances.

Relevance to Harnesses / Meta-Harnesses

SWE-Compass is itself a meta-harness: it wraps heterogeneous agentic frameworks (SWE-Agent, Claude Code) under a unified, controlled evaluation harness with reproducible Docker environments, fixed budgets, and task-type-specific metric dispatching — exactly the architecture pattern of a meta-harness that standardizes execution across diverse agent backends. The five-stage construction pipeline (including automated environment build, expert repair loops, and multi-round validation) demonstrates how a harness must be engineered to reliably orchestrate agent runs at scale across diverse software ecosystems. For researchers building harnesses, the environment-build failure rate (~98% automated failure) is a key practical data point: reliable harness infrastructure for real repositories demands significant human-in-the-loop repair capacity, not just automation.