Skip to content

Understanding Agent-Based Patching of Compiler Missed Optimizations

🕒 Published (v1): 2026-07-02 16:12 UTC · Source: Arxiv · link

Why this paper was selected

Agent-based patching of compiler missed optimizations; novel agentic coding application

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically studies whether LLM-based coding agents can patch compiler missed optimizations in LLVM in a way that generalizes to the developer-intended scope, not just the reported test case. Agents frequently fix the given example but misalign with the intended optimization scope—either under-generalizing or partially overlapping. Historical-knowledge augmentation via retrieval (RAG) and distillation of prior LLVM PRs improves developer-aligned generalization.

Problem

Patching a compiler missed optimization is fundamentally different from conventional bug fixing: success requires inferring a generalized optimization rule that applies beyond the single reported IR fragment, not merely satisfying the initial test case. Existing LLM agent evaluations focus on passing existing tests, which is an insufficient proxy for developer intent in this domain. No prior benchmark or framework measured optimization-scope alignment between agent-generated and developer-written patches.

Method

Benchmark: 43 verified LLVM missed-optimization instances drawn from closed, fixed GitHub issues. Each instance includes an initial test case (provided to the agent), a golden patch (from the fixing commit), and golden test cases (extracted from the fixing commit's regression tests, withheld from the agent). Instances span InstCombine (32), SimplifyCFG (4), ValueTracking (3), ConstraintElimination (2), and InstructionSimplify (2).

Agent harness: A lightweight harness connects a model to an issue-specific LLVM workspace at the pre-fix commit via a controlled tool interface (issue_info, view_source, get_langref, apply_code, build_and_check, lit_check, show_diff, submit_patch). The agent runs a ReAct loop (up to 50 steps) and a patch is accepted only if it passes Alive2 semantic verification and llvm-mca cost checks.

Scope evaluation: Agent patch scope \(A = S(T_A)\) is compared to golden scope \(G = S(T_G)\) using: - Golden tests to detect under-generalization (\(A \subset G\)) - Fuzzing (1,000 alive-mutate mutants + 60 LLM-synthesized IR programs per patch pair) to detect over-generalization (\(G \subset A\))

This yields four categories: \(A \subset G\) (under-generalizes), \(A \bowtie G\) (partial overlap), \(G \subset A\) (over-generalizes), \(A \sim G\) (aligned).

Historical-knowledge augmentation (RQ3–RQ4): - RAG: Retrieves similar source-to-target IR transformation pairs from historical LLVM optimization PRs as concrete precedents. - Distillation: Summarizes recurring component-level generalization principles from prior PRs as compact background knowledge injected into the agent prompt.

Models evaluated: GPT-5.5, DeepSeek-V4-Pro, Qwen3.5-Plus, Kimi K2.5, using mini-SWE-agent as the coding-agent framework.

Key Contributions

  • Formulates compiler missed-optimization patching as a generalization-sensitive task where scope alignment with developer intent is the primary success criterion.
  • A benchmark of 43 verified LLVM missed-optimization instances with scope-based evaluation infrastructure (golden tests + dual fuzzing strategy).
  • Empirical characterization of how agents misalign: frequently fix the initial test case but diverge from intended scope.
  • Shows that generic generalization instructions provide limited and sometimes harmful benefit.
  • Proposes retrieval- and distillation-based historical-knowledge augmentation from prior LLVM optimization PRs, improving developer-aligned generalization and yielding practical IR optimization improvements.

Results

  • Patch generation (RQ1): GPT-5.5 generates a valid patch (passing the initial test) for 74% (32/43) of issues; other models range 26–32 applied. Among successful patches, roughly half achieve \(A \sim G\); the remainder exhibit under-generalization (\(A \subset G\)) or partial overlap (\(A \bowtie G\)).
  • Generic generalization instruction (RQ2): Adding the instruction "handle more general patterns" yields mixed or negative effects:
  • GPT-5.5: \(A \sim G\) drops from 18 → 17; failures increase 11 → 13.
  • DeepSeek-V4-Pro: \(A \sim G\) drops from 14 → 10; \(A \subset G\) increases 5 → 8.
  • Generic guidance does not reliably align patches with developer intent and can worsen alignment.
  • Historical-knowledge augmentation (RQ3–RQ4): RAG and distillation improve alignment across models and yield practical benefits when applied to real-world IR optimization (specific numbers are reported in the paper but fall in the truncated section).

Limitations

  • Benchmark is modest in size (43 instances) due to strict validation requirements (reproducible initial test, identifiable fix, recoverable golden tests).
  • Scope evaluation is proxy-based: golden tests are a lower-bound sample of intended scope; fuzzing cannot exhaustively enumerate \(A\) and \(G\), so \(A \sim G\) is "no observed difference," not proven equivalence.
  • Agent-only fuzz cases that over-generalize may in fact represent valid broader optimizations not yet tested by developers—the framework cannot distinguish valid from invalid over-generalization.
  • Evaluation is restricted to LLVM (primarily InstCombine); generalizability to other compilers or optimization frameworks is unverified.
  • Cost constraints limited LLM-based fuzzing to 60 test cases per patch pair.

Relevance to Agentic AI / LLM Agents

This work directly addresses a critical gap in agentic coding research: test-passing is an insufficient success criterion when the task requires inferring developer intent beyond the given examples, a challenge that extends well beyond compiler work to any domain where agents must generalize rules from exemplars (API misuse fixes, security patches, refactorings). The finding that generic "be more general" instructions fail—and that retrieval and distillation of domain-specific historical experience are more effective—has broad implications for how agentic systems should be augmented with institutional or corpus knowledge. The scope-alignment evaluation framework (golden tests + adversarial fuzzing to characterize \(A \bowtie G\)) also offers a reusable methodology for measuring behavioral alignment between agent and human patches in other structured program-transformation settings.