SWERank: Software Issue Localization with Code Ranking

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; SWERank: code ranking for software issue localization in coding agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWERank is a retrieve-and-rerank framework for software issue localization that outperforms costly LLM-based agent systems (e.g., SWE-Agent with Claude-3.5) at a fraction of the cost. It pairs a bi-encoder retriever (SWERankEmbed) with a listwise LLM reranker (SWERankLLM), both trained on SWELoc, a new 67K+ instance dataset of real GitHub issue/patch pairs. State-of-the-art results are achieved on SWE-Bench-Lite and LocBench across file, module, and function granularities.

Problem

LLM-based agentic localization systems (SWE-Agent, OpenHands, LocAgent) require 7–10 multi-step reasoning rounds and cost ~$0.66 per example with Claude-3.5, while being brittle to intermediate failures. Conventional code ranking models trained on NL-to-code or code-to-code retrieval are mismatched for issue localization: bug reports average 460 tokens (vs. 12 tokens for standard code search queries) and describe erroneous behavior rather than desired functionality. No large-scale, task-specific training dataset for issue localization existed.

Method

Two-stage retrieve-and-rerank:

1. SWERankEmbed (bi-encoder retriever): Initialized from CodeRankEmbed (137M) or GTE-Qwen2-7B-Instruct (7B). Trained with InfoNCE contrastive loss: $\(L_{CL} = -\log \frac{\exp(h_i \cdot h_i^+)}{\sum_{h_k \in (H_B \cup H)} \exp(h_i \cdot h_k)}\)$ where \(H_B\) contains in-batch negatives and \(H\) contains \(M=15\) hard negatives per instance mined by similarity to the query.

2. SWERankLLM (listwise reranker): An instruction-tuned LLM (7B or 32B) that takes the issue and top-K retrieved candidates with assigned identifiers. Because SWELoc lacks a full negative ranking, training maximizes only the likelihood of the single positive identifier \(I_i^+\): $\(L_{LM} = -\log P_\theta(I_i^+ | x)\)$ The EOS token is omitted after \(I_i^+\) to preserve full-list generation capability at inference.

SWELoc dataset construction: (1) Select 3,387 Python repositories from top-11K PyPI packages, excluding SWE-Bench and LocBench repos. (2) Collect PRs resolving linked GitHub issues with test modifications → 67,341 initial (PR, codebase) pairs. (3) Apply top-\(K\) consistency filtering (retain positive only if it ranks within top \(K=20\) by a pretrained embedder) and mine \(M=15\) hard negatives per instance.

Key Contributions

• SWERank retrieve-and-rerank framework with task-specific bi-encoder retriever and listwise LLM reranker
• Modified listwise reranking training objective that requires only a single positive label (no full negative ranking needed)
• SWELoc: 67,341-instance GitHub issue localization dataset with consistency filtering and hard negative mining
• SOTA on SWE-Bench-Lite and LocBench at all granularities (file, module, function) using only open-source models
• Demonstration that SWELoc generalizes: finetuning third-party retrievers and rerankers on SWELoc consistently improves their localization performance

Results

SWE-Bench-Lite (274 examples): - SWERankEmbed-Small (137M): File Acc@1 = 66.42%, surpassing all prior 7B embedding baselines and CodeRankEmbed (52.55%) - SWERankEmbed-Large (7B): File Acc@1 = 72.63%, Function Acc@10 = 82.12%, exceeding LocAgent+Claude-3.5 (77.74% / 77.37%) on function Acc@10 - SWERankLLM-Large (32B) + SWERankEmbed-Large: File Acc@1 = 83.21%, Function Acc@10 = 88.69% — best among open-source systems, surpassing GPT-4.1 reranker (82.12% / 87.96%) on File Acc@1

LocBench (~560 examples): - SWERankEmbed-Large: File Acc@5 = 82.14%, Acc@10 = 86.96%, vs. LocAgent+Claude-3.5 (83.39% / 86.07%) - SWERankLLM (not fully shown) further improves on these numbers

Cost: Agent-based methods (Claude-3.5) cost ~$0.66/example; SWERank achieves superior or matching accuracy at substantially lower cost (qualitative advantage, exact SWERank cost not specified in the text).

Limitations

• SWELoc is exclusively Python; generalization to other languages not evaluated
• Exact per-example inference cost of SWERankLLM is not quantified, making the cost comparison imprecise
• The modified listwise training objective (predicting only the top-1 positive) is a proxy for full ranking supervision; the gap vs. full-supervision reranking is not directly ablated
• Hard negatives are drawn from the same repository, which may not capture cross-repository confusion patterns
• LocBench comparisons are limited to a subset of baselines due to dataset scale

Relevance to Agentic AI / LLM Agents

SWERank directly challenges the dominant paradigm of using multi-step LLM agents for code localization — a foundational sub-task in automated software engineering pipelines (SWE-Bench, Devin-style systems). By reframing agentic exploration as a single-shot ranking problem, it demonstrates that task-specific retrieval training can match or beat 7–10-round agent reasoning with closed-source models like Claude-3.5, at orders-of-magnitude lower cost. For agent researchers, the work highlights a general design pattern: replacing expensive iterative agent loops with purpose-built IR components wherever the problem permits a ranking formulation. SWELoc also serves as a reproducible training resource for researchers building agentic coding assistants that need efficient, open-source localization modules.