A Study on the Impact of Fault localization Granularity for Repository-Scale Code Repair Tasks¶

🕒 Published (v1): 2026-03-31 19:16 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper empirically investigates whether fault localization (FL) granularity—line, function, or file level—affects downstream patch generation in repository-scale automated program repair (APR), using a "perfect localization" assumption to isolate granularity as the sole variable. On SWE-Bench-Mini with Qwen3-Coder-30B inside the Agentless framework, function-level granularity achieves the highest resolve rate (45.6%) with lowest variance, though the optimal granularity appears task-dependent when stratified by difficulty.

Problem¶

Repository-scale APR pipelines consistently split fault localization and patch generation into separate phases, yet no prior work has empirically justified which granularity of localization context (line, function, or file) best enables a repair agent to generate correct patches. Existing frameworks treat granularity as an implementation detail rather than a principled design variable, and no study has isolated this question from localization accuracy errors by testing under perfect (ground-truth) localization at repository scale.

Method¶

The authors modify the Agentless framework by replacing its localization phase with a script that extracts ground-truth localization directly from the diff of the solution pull request in SWE-Bench-Mini (50 instances, django and sphinx-doc repos). Three context formats are injected into the repair prompt: - Line-level: exact changed lines ± 10 surrounding lines - Function-level: full function/class bodies containing any changed line ± 10 surrounding lines - File-level: entire files containing changed functions

Agentless' patch validation/re-ranking phase is also bypassed (first patch accepted) to prevent re-ranking from confounding the granularity signal. Qwen3-Coder-30B served via vLLM v0.15.1 acts as the repair model. Each granularity is evaluated across 10 independent trials, and Friedman test is used to assess statistical significance.

Key Contributions¶

First empirical study of FL granularity under the perfect localization assumption at repository scale (SWE-Bench-Mini).
Demonstration that function-level granularity yields the highest overall resolve rate and lowest trial-to-trial variance among the three levels tested.
Evidence that the resolve rate ranking across granularities shifts with instance difficulty, suggesting no single granularity is universally optimal.
Argument and framework for using perfect-localization baselines before developing new FL methods, to first confirm any granularity is actionable by the repair agent.

Results¶

Function-level: 45.6% resolve rate (\(\sigma = 1.5\))
Line-level: 43.6% resolve rate (\(\sigma = 2.6\))
File-level: 42.6% resolve rate (\(\sigma = 1.6\))
Friedman test: \(\chi^2 = 12.8\), \(\rho = 0.0017\) (statistically significant differences)
Even at perfect localization, best resolve rate is ~45.6%—over 20 percentage points below state-of-the-art on SWE-Bench-Mini (Claude Sonnet 4.5 achieves >65%)
Difficulty-stratified results: granularity ranking is not stable across easy/medium/hard tiers, indicating task-dependent optima

Limitations¶

Small dataset: 50 instances from only 2 repositories (django, sphinx-doc), limiting generalizability across the 12 repos in SWE-Bench Verified
Single repair model (Qwen3-Coder-30B); hypothesis that optimal granularity aligns with training granularity is untested across models
"Perfect localization" is approximated via PR diffs, which may omit contextually relevant but unmodified code
No test-time scaling, agentic tool use, or prompt engineering—pipeline is deliberately simplified, so results may not reflect full system performance
Study is not designed to improve state-of-the-art; findings are preliminary proof-of-concept

Relevance to Harnesses / Meta-Harnesses¶

This paper directly probes a core design decision in multi-phase APR harnesses: how to structure the information hand-off between a localization phase and a repair agent phase. By instrumenting Agentless as a controlled harness and surgically replacing one phase with ground-truth data, the authors demonstrate a meta-harness evaluation methodology—isolating individual pipeline stages to measure their contribution independently. The finding that granularity (i.e., context engineering for inter-agent communication) significantly affects downstream resolve rate is immediately actionable for anyone designing orchestration harnesses that decompose repair into locate→patch→validate steps. The call for a more systematic prompt-template search space exploration also points toward meta-harness concerns: how to configure, sweep, and evaluate the parameters of an agentic pipeline without conflating multiple confounders.