Skip to content

SHERLOC: Structured Diagnostic Localization for Code Repair Agents

🕒 Published (v1): 2026-06-23 17:05 UTC · Source: Arxiv · link

Why this paper was selected

NVIDIA+TU Darmstadt; structured localization cuts agent budget waste in half for repo repair

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SHERLOC is a training-free fault localization framework that pairs a reasoning LLM with a compact, structured tool suite to identify buggy code regions and emit five-field diagnostic findings (root cause, solution idea, dependencies, etc.) rather than bare file paths. On SWE-Bench Lite and Verified it reaches state-of-the-art file-level localization, and injecting its findings into downstream repair agents yields +5.95 pp resolve rate while cutting total tokens by 23.1%.

Problem

LLM-based code repair agents spend ~48% of their interaction budget (≈18.5 turns, >320k tokens per instance) just locating faults before writing the first patch. Existing localization methods either return ranked file lists (no diagnostic context) or produce free-form reports via multi-agent debate or supervised fine-tuning—neither providing the structured, actionable root-cause context a repair agent needs, nor doing so without additional training.

Method

SHERLOC couples a single reasoning LLM with four deterministic, schema-structured tools: View File (file inspection with fuzzy path matching), Codebase Search (repository-wide literal search), Repository Tree (filtered directory hierarchy), and Connected Tree (import-graph navigation). The LLM runs an iterative loop of at most 20 turns, alternating reasoning/action-selection → action parsing → tool execution → observation accumulation. No external memory store is used; state is the conversation history. On termination, the model emits (a) file/line-span locations and (b) a structured diagnostic finding with five fields: location explanation, root cause, solution idea, dependencies, and testing impact. Self-recovery handles context truncation, loop detection, malformed tool calls, and forced final-turn synthesis when the budget is exhausted. No fine-tuning, RL, or multi-agent orchestration is used. Findings are injected into downstream repair agents (SWE-Agent, OpenHands) via a quality filter: a GPT-5.2 judge scores each finding 1–5 on root-cause correctness, location accuracy, and solution actionability; findings below a composite threshold of 4.0 fall back to baseline agent execution.

Key Contributions

  • Training-free, single-LLM localization framework with a minimal fixed tool suite and lightweight self-recovery, achieving SOTA file-level accuracy without task-specific training.
  • Structured five-field diagnostic findings per predicted location (not just file paths), providing actionable repair context.
  • Structure-agnostic chunk-level localization metrics over \((start, end)\) line-number spans—coverage recall, precision, and average tightness—decoupled from syntactic units (functions, classes).
  • Implicit-knowledge controls (path masking, repository shuffling) to isolate tool-assisted reasoning from pretraining familiarity with public SWE-Bench repositories.
  • Dual-axis downstream transfer: +5.95 pp average resolve rate and 36.7%/23.1% reduction in localization/total tokens across five repair backbones and two frameworks.

Results

  • SWE-Bench Lite (file-level accuracy@1): 84.33 ± 0.72% (Qwen3-235B), surpassing OrcaLoca/Claude 3.5 Sonnet (83.33%) and SWERank/Qwen2.5-32B-FT (83.21%).
  • SWE-Bench Verified (file-level recall@1): 81.27 ± 1.16% (Qwen3-235B), +13.2 pp over prior SOTA (RepoSearcher/ToolTrain-32B-FT, 68.03%).
  • At matched ~30B scale: Qwen3-30B reaches 75.07% recall@1, outperforming fine-tuned ToolTrain-32B (+7.0 pp) and all non-fine-tuned 32B baselines.
  • Cross-family transfer: DeepSeek-V3-0324 achieves 79.53% recall@1, 1.7 pp below the Qwen3-235B ceiling.
  • Chunk-level (SWE-Bench Verified, Qwen3-235B): 41.28% coverage recall, 53.47% precision.
  • Downstream resolve rate (average across 10 model-framework pairs): +5.95 pp over baseline; localization tokens −36.7%, total tokens −23.1%.
  • Finding quality: 54% of 499 judged findings score >4.0 (Very High); Very High quality findings resolve at 75.9% vs. 61.6% baseline; Low quality findings underperform the baseline.
  • Masking control: Removing explicit file paths from issue text reduces recall by only 1.3 pp (81.27% → 79.96%), showing active reasoning drives most of the gain.

Limitations

  • Chunk-level metrics cannot be compared to prior systems, which do not report span-level scores.
  • Transfer gains are quality-mediated: low-quality findings (11% of instances) actively hurt downstream resolve rate below baseline, requiring a judge filter that adds inference cost.
  • Optimal quality-filter threshold (≥4.0) was determined with GPT-5.2 as judge, introducing a dependency on a proprietary model for quality gating.
  • DeepSeek-R1-0528 exhibits 41% fewer tool calls than Qwen3-235B due to prompt/system-role token differences, suggesting SHERLOC's performance is sensitive to backbone tool-use compliance.
  • Gains plateau for already-strong localizers (e.g., MiniMax agents), where quality filtering rather than injecting all findings is the effective intervention.
  • Evaluated only on SWE-Bench Lite/Verified; generalization to other repository-level repair benchmarks is not demonstrated.

Relevance to Harnesses / Meta-Harnesses

SHERLOC is essentially an interface-design paper for multi-stage agent pipelines: it shows that the information contract between a localization stage and a repair stage matters enormously, and that replacing bare file paths with a five-field structured finding (root cause, fix idea, cross-file dependencies, etc.) is worth +5.95 pp resolve rate at lower token cost. For harness builders, the key insight is that richer, semantically typed inter-agent handoffs outperform simply giving the repair agent more context or compute — the structured intermediate representation does real work. It also validates the decompose-then-specialize pattern (separate localization from repair) as a harness-level architectural choice, not just a research convenience, since the gains hold across multiple downstream repair agents without retraining either component. Concretely, if you're designing a meta-harness that routes tasks through specialist agents, SHERLOC gives you a blueprint for what the handoff schema should contain and evidence that schema quality is a first-class performance lever.