Skip to content

How and Why Agents Can Identify Bug-Introducing Commits

🕒 Published (v1): 2026-03-31 07:48 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM-based agentic workflows (SZZ-Agent and the simpler Simple-SZZ-Agent) substantially advance the 20-year-old problem of identifying bug-introducing commits (BICs), raising F1-score from 0.64 (prior SOTA) to 0.81 on the canonical Linux kernel dataset. The key mechanistic finding is that agents spontaneously distill fix-commit semantics into short grep patterns to search large candidate sets, making binary search unnecessary. Surprisingly, a simpler single-stage harness outperforms a carefully engineered multi-stage pipeline at 43% lower token cost.

Problem

Two decades of SZZ-algorithm refinements have produced only ~10 percentage points of cumulative F1 improvement (0.54 → 0.64) on the Linux kernel BIC dataset. All existing approaches—annotation-graph filters, refactoring-aware filters, graph neural networks, and LLM-based re-rankers (LLM4SZZ, ISSTA 2025 SOTA)—remain limited by the core SZZ assumption that deleted fix-commit lines are the same lines that introduced the bug, which fails when a fix adds code rather than removing it.

Method

SZZ-Agent is a two-stage agentic pipeline: 1. Stage 1: Run standard SZZ to produce candidate commits from deleted-line annotations; an agent (Claude Opus 4.5 via Claude Code) receives the fix commit message + diff and each candidate's message + diff, then selects the BIC or abstains. 2. Stage 2: Activated on abstention or when SZZ finds no candidates; collects full file-commit histories, binary-searches over them (agent reads full file contents at midpoint commits to determine whether the bug is already present), then switches to direct candidate selection once the window narrows to a tunable threshold (default 33).

An ablation varying the threshold found that \(\text{threshold} = \infty\) (skip binary search entirely) achieves the best F1 at lowest cost, motivating Simple-SZZ-Agent: collect all file-commit histories into one candidate set, provide the agent with grep/glob/file-read tools, and instruct it to identify the BIC directly—no SZZ filtering, no binary search. Behavioral analysis reveals the emergent agent strategy: compress the fix commit message and diff into short string patterns, grep the candidate set for matches, read and evaluate results, and iterate until the BIC is found.

Key Contributions

  • SZZ-Agent: Two-stage pipeline (SZZ filtering → binary search → direct selection) raising F1 by 12–13 pp over LLM4SZZ on three C/C++ and Java datasets; five ablation studies characterize why it works.
  • Simple-SZZ-Agent: Single-stage agent that matches or exceeds SZZ-Agent on all datasets at 43% lower token cost, with resource consumption nearly independent of candidate set size.
  • Mechanistic explanation: Agent performance stems from the agentic approach itself—not data leakage (validated on post-cutoff DS_LINUX-26) and not LLM backbone quality (upgrading LLM4SZZ to Claude Opus 4.5 gives ≤0.65 F1; SZZ-Agent with the same backbone gives 0.77).
  • Grep-pattern insight: Agents compress semantic bug understanding into greppable string patterns, opening avenues for bug detection, root cause analysis, and automated repair.

Results

  • SZZ-Agent vs. LLM4SZZ on established datasets: +13 pp F1 on DS_LINUX (0.77 vs. 0.64), +12 pp on DS_GITHUB-c (0.75 vs. 0.63), +12 pp on DS_GITHUB-j (0.67 vs. 0.55); all significant at \(p < 0.01\) with large effect sizes (except DS_GITHUB-j \(p = 0.013\))
  • Simple-SZZ-Agent vs. SZZ-Agent: +4 pp on DS_LINUX (0.81 vs. 0.77), parity on DS_GITHUB-c (0.75), +8 pp on DS_GITHUB-j (0.75 vs. 0.67)
  • Post-training-cutoff validation (DS_LINUX-26, Sep 2025–Jan 2026): SZZ-Agent 0.81 F1 vs. LLM4SZZ 0.62—rules out memorization
  • LLM backbone ablation: LLM4SZZ with gpt-5.2 or Claude Opus 4.5 reaches at most 0.65 F1 vs. 0.77 for SZZ-Agent with the same Claude Opus 4.5
  • Context ablation (DS_LINUX-26): fix diff only → 0.73 F1; fix message only → 0.77; both → 0.81; neither → 0.48 (below SZZ baseline of 0.56)
  • Stage ablation: Stage 1 alone → 0.48 F1 at $0.18/fix; Stage 2 alone → 0.76 at $1.20/fix; combined → 0.81 at $0.67/fix
  • Threshold ablation: F1 stable across threshold 8–∞ (0.76–0.82); cost drops from $0.68 (threshold 8) to $0.33 (\(\infty\), best F1)

Limitations

  • Evaluation is budget-constrained: DS_LINUX (200 samples), DS_GITHUB-c (100), DS_GITHUB-j (75); results may not generalize to full-dataset scale
  • Per-fix cost of $0.33–$0.67 is likely prohibitive for repository-wide BIC scanning
  • Strategy analysis is behavioral (tool-call traces), not formal; grep-pattern extraction is inferred, not instrumentally verified
  • Open-source LLM and open-source agent experiments (§4.3.4) are truncated in the available text
  • Downstream utility of extracted grep patterns for bug detection or repair is proposed but not evaluated

Relevance to Harnesses / Meta-Harnesses

This paper is directly about harness design: the authors build Claude Code (with 30 built-in tools: bash, grep, glob, file read, sub-agent spawning) into two explicit agentic pipelines and systematically ablate their architecture. The central meta-harness finding is that a simpler, single-stage harness (Simple-SZZ-Agent) outperforms a carefully engineered multi-stage one—an emergent agent strategy inside the harness substitutes for explicit workflow engineering, which challenges the common assumption that more structured pipelines are always better. The five-ablation evaluation methodology (data leakage, backbone, context, stage contribution, hyperparameter) is a reusable template for rigorous harness evaluation applicable to any multi-step agentic system.