Skip to content

SWE-QA: Can Language Models Answer Repository-level Code Questions?

🕒 Published (v1): 2025-09-18 05:25 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWE-QA is a repository-level code QA benchmark of 720 human-verified question-answer pairs across 15 Python repositories, requiring cross-file, multi-hop reasoning. It evaluates six LLMs under five context augmentation strategies—direct prompting, two RAG variants, and two agent frameworks—finding that agent frameworks substantially outperform RAG, with the best result at 70.79/100 (GPT-5.1 + OpenHands).

Problem

Existing code QA benchmarks (CoSQA, CodeQA, CodeRepoQA) target isolated functions, snippets, or APIs, failing to capture repository-level complexity: cross-file dependencies, architectural reasoning, multi-hop dependency chains, and design rationale. Prior repository-level attempts (CoreQA, Spyder-CodeQA) lack human verification, multi-hop questions, or comprehensive taxonomic coverage.

Method

Benchmark construction follows a four-stage pipeline: 1. Seed collection: 77,100 GitHub issues crawled from 12 SWE-bench repos, filtered to 41,955 substantive issues, parsed by LLM to yield 127,415 questions; two annotators built a two-level taxonomy (What/Why/Where/How Ă— 12 fine-grained intentions). 2. Question instantiation: tree-sitter parses each repo into a structural graph; LLM fills seed templates using focal-element subgraphs to produce context-specific questions. 3. Answer collection: RAG pipeline (semantic similarity + structural dependency index) retrieves code context; strong LLM generates grounded answers, with 4 human experts assisting. 4. Validation: expert pairs independently verify each answer; disagreements resolved by a third expert; quality filtering enforces per-repo balance of 48 pairs across the four Level-1 types.

Evaluation uses LLM-as-Judge (Claude Sonnet 4.5) scoring five dimensions (correctness, completeness, relevance, clarity, coherence) on a 100-point scale, with human evaluation used to validate judge reliability.

Key Contributions

  • SWE-QA benchmark: 720 human-verified QA pairs from 15 diverse Python repos (13,300 files, 142,404 functions, 3.4M+ LOC); avg. answer length 266.64 words; 77.6% require cross-file reasoning; avg. reasoning chain depth 4.72.
  • Extensible QA generation pipeline: semi-automatic (tree-sitter + LLM + human filter) pipeline applicable to any open-source repo.
  • Systematic evaluation: six LLMs Ă— five context strategies (direct, sliding-window RAG, function-chunking RAG, SWE-agent, OpenHands) plus two commercial baselines (Cursor, Tongyi Lingma).
  • Two-level taxonomy: 4 primary interrogatives Ă— 12 fine-grained intentions grounded in 127,415 real developer questions.

Results

  • Best overall: GPT-5.1 + OpenHands = 70.79/100; GLM-4.6 + OpenHands = 70.15; Cursor = 70.66.
  • Direct prompting baseline: Kimi K2 = 51.47; Qwen3-30B = 50.80; Qwen3-480B = 54.61; GLM-4.6 = 56.64.
  • RAG gains: Sliding Window RAG adds ~+11–14 points over direct prompting for most models.
  • Agent gains over RAG: SWE-agent/OpenHands further adds ~+5–16 points; Kimi K2 reaches 67.72 with SWE-agent.
  • Smaller models: Qwen3-30B benefits less; agent frameworks sometimes underperform RAG for this model.
  • Taxonomy gaps: Why questions (avg. 69.77) and "Design rationale" (71.48) are easiest; "Architecture exploration" (61.84) and Where questions (avg. 66.76) are hardest.
  • Human–judge agreement: human evaluation confirms high correlation with LLM-as-Judge scores.

Limitations

  • Restricted to Python repositories; generalization to other languages is unverified.
  • Only 720 QA pairs (48 per repo), which may underrepresent the long tail of repository-specific patterns.
  • LLM-as-Judge scoring may carry residual model bias despite judge–candidate separation and answer anonymization.
  • Smaller models (e.g., Qwen3-30B) show diminishing returns from agent frameworks, and the cause is not fully analyzed.
  • "Pylint" and other structurally complex repos show notably lower performance; repo-level variance is not systematically explained.
  • Reference answers are themselves LLM-assisted, introducing possible systematic blind spots despite human review.

Relevance to Harnesses / Meta-Harnesses

SWE-QA directly benchmarks the agentic harness tier of code understanding: it pits iterative, tool-using agent frameworks (OpenHands, SWE-agent) against static RAG pipelines and reports the performance gap as a function of harness architecture, not model capability alone. The benchmark construction pipeline is itself a semi-automated meta-harness—tree-sitter static analysis, LLM question instantiation, RAG answer generation, and human-in-the-loop validation—that can be re-run on any repository to extend the benchmark. For researchers studying harness design, the results quantify when adding a planning/tool-invocation layer atop retrieval yields meaningful gains (\(\sim\)+5–16 points) and when it doesn't (small-capacity models), making SWE-QA a diagnostic tool for comparing harness strategies on realistic, multi-hop software engineering tasks.