AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning

🕒 Published (v1): 2026-06-18 15:35 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoPass is a four-agent LLM framework for compiler performance tuning that grounds optimization decisions in LLVM-internal signals (IR snapshots, optimization remarks) and iterative runtime profiling feedback. Operating inference-only with no task-specific fine-tuning, it achieves geometric-mean speedups of 1.043× and 1.117× over LLVM -O3 on x86-64 and ARM64, respectively, outperforming PGO variants and search-based autotuning under the same evaluation budget.

Problem

Existing LLM approaches for compiler optimization target static, deterministic objectives such as code size; optimizing runtime performance is harder because it depends on complex microarchitectural interactions and noisy measurements. Without access to compiler-internal signals, an LLM cannot reliably assess whether its proposed pass-pipeline actually improves execution speed. Profile-guided optimization (PGO) remains conservative when branch evidence is noisy, while black-box search (e.g., OpenTuner) cannot discover strong configurations under tight compile-and-run budgets typical of practical deployment.

Method

AutoPass is a four-agent pipeline orchestrated via CrewAI with DeepSeek-V3.2 as the LLM backend.

1. Score Agent — scans the source tree, constructs an inter-procedural callgraph, and assigns each function a priority score from static IR features (\(\#\)BasicBlocks, \(\#\)Loops, \(\#\)Calls, \(\#\)CondBranch). Trivial and I/O-bound functions are pruned so downstream agents focus on high-impact kernels.

2. Analysis Agent — for each selected function, runs semantic hint inference (from symbol names and metadata) and remark-guided structural analysis of LLVM IR combined with compiler diagnostics (-Rpass, -Rpass-missed, -Rpass-analysis). Outputs a normalized JSON containing semantic hints and categorized missed-opportunity remarks.

3. Reasoning Agent — receives the Analysis Agent's JSON summary plus hardware target info, then proposes a modified pass pipeline starting from the -O3 reference. In subsequent iterations it incorporates runtime feedback to prune ineffective passes, adjust parameters (e.g., unroll factors, inlining thresholds), and address remaining bottlenecks. A deterministic repair-and-validation stage corrects malformed syntax and maps hallucinated pass names to the nearest valid pass; candidates failing schema correctness, pass-membership, parameter-range, or compilation checks are rejected before execution.

4. Evaluation Agent — compiles and runs each validated candidate three times; accepts if mean runtime \(t(P^{(t)}) < t(P^\star)\), otherwise rolls back to \(P^\star\). Summarizes observed latency deltas and updated remarks as feedback for the next Reasoning round. The loop terminates after a fixed iteration budget; the final output is \(P^\star\) only if it beats -O3, otherwise -O3 is returned.

The search space spans 74 LLVM optimization passes with sequences up to 107 passes.

Key Contributions

• A multi-agent harness that exposes compiler-internal state (IR + diagnostic remarks) to an LLM rather than treating the compiler as a black box.
• A feedback-driven loop integrating structured pipeline editing, deterministic repair/validation, runtime measurement, and rollback to enforce robustness.
• Empirical demonstration that an inference-only, training-free LLM agent can outperform PGO methods and constrained black-box search on both pass ordering and parameter tuning.

Results

• Geometric-mean speedup of 1.043× over LLVM -O3 on x86-64 (Intel Core i9) across the full benchmark suite.
• Geometric-mean speedup of 1.117× over LLVM -O3 on ARM64 (Cortex-A76 / Raspberry Pi 5).
• On motivating benchmarks (QSort + BitCount), average speedup of 1.259× over -O3.
• Outperforms Instrumented PGO, AutoFDO, CSSPGO, and OpenTuner (all given the same 3-iteration budget).
• OpenTuner (black-box, same budget) fails to find improvements on both motivating benchmarks; PGO variants regress on BitCount due to hot-path misclassification.

Limitations

• Only three feedback iterations are used; performance under larger budgets is not reported in the available text.
• Context-window size (128K tokens for DeepSeek-V3.2) requires aggressive function filtering by the Score Agent, potentially missing cross-function interactions.
• Evaluation is limited to LLVM/Clang 17.0.6; generalization to GCC or other compiler IR formats is not demonstrated.
• Candidate pass pipelines are restricted to passes in the initial -O3 set, excluding passes disabled at -O3 from consideration.
• Noisy runtime measurements are partially controlled (mean of 3–5 runs, Turbo Boost disabled) but hardware variability on ARM64 embedded device (8 GB RAM) is not formally characterized.
• Comparison excludes learned-policy baselines (ACPO, Autophase, CompilerGym) due to unavailable weights or differing objectives, limiting scope of comparison.

Relevance to Harnesses / Meta-Harnesses

AutoPass is a textbook example of a domain-specific multi-agent harness: each of its four agents has a tightly scoped, bounded role, and their outputs pass through deterministic validation layers before being forwarded downstream — a pattern central to reliable meta-harness design. The repair-and-validation stage (syntax correction, pass-name mapping, schema checking) directly addresses the standard harness problem of making unreliable LLM outputs safe to act on without human review. The explicit feedback accumulator and rollback mechanism mirror patterns seen in software-engineering meta-harnesses (e.g., SWE-agent) where environment state must be checkpointed to survive failed iterations. For harness researchers, AutoPass illustrates how to close the loop between an LLM reasoning core and a stateful external environment (the LLVM toolchain) using evidence injection and structured intermediate representations rather than raw text.