Skip to content

cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution

🕒 Published (v1): 2025-12-18 12:34 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

cuPilot is a multi-agent evolutionary framework for automated CUDA kernel optimization that introduces strategy as an intermediate semantic representation, decoupling high-level evolutionary search from low-level kernel code. It achieves an average 3.09Ă— speedup over PyTorch across 100 KernelBench kernels, substantially outperforming the prior AI CUDA Engineer baseline.

Problem

Existing LLM+evolutionary-algorithm approaches to CUDA kernel optimization suffer from three representational mismatches: (1) crossover prompting operates directly on kernel code, forcing LLMs to traverse a long reasoning chain (strategy identification → combination → synthesis) that degrades as kernel complexity grows—success rate drops to 1% for complex parents; (2) fitness functions expose only raw performance, providing no semantic guidance for bottleneck identification; (3) population initialization draws from sparse kernel code, limiting strategy diversity and causing premature convergence.

Method

cuPilot introduces strategy (a structured, natural-language description of optimization techniques) as a semantic intermediate between evolutionary search and kernel code. The framework comprises four agents:

  • SCE Manager: runs strategy-level crossover and elitism/tournament selection, operating entirely in strategy space.
  • Strategy Translator: applies a strategy to a kernel (StrategyApplication) then back-extracts the realized strategy from the revised kernel (StrategyAlignment), maintaining strategy-kernel consistency.
  • Kernel Revisor: handles syntax/functional correction and Nsight Compute profiling.
  • Roofline Prophet: classifies each kernel as compute-bound, memory-bound, or middle-zone using the GPU roofline model, then injects hardware-specific prompts (e.g., prioritize SM Throughput for compute-bound; DRAM/L2 for memory-bound) into all three evolutionary agents.

Population initialization uses RAG over a historical database of (initial kernel, optimized kernel, profiling metrics) tuples, retrieved by kernel similarity to bootstrap strategy generation. The SCE algorithm iterates over epochs (outer) and generations (inner), with tournament+elitism selection crossing strategies rather than codes, then translating crossed strategies to kernels in parallel.

Key Contributions

  • Strategy-Coordinated Evolution (SCE) algorithm: decouples crossover into strategy-level recombination + strategy-to-kernel translation, eliminating the long reasoning chain that causes failure in complex kernel crossover.
  • Roofline-guided prompting: positions kernels on the GPU roofline model to determine bound type, then provides targeted profiling metrics and strategy suggestions, replacing uninformative global performance signals.
  • Strategy-level population initialization via RAG: maintains an external strategy pool augmented with historical evolution data; retrieves similar past strategies to bootstrap new kernel optimization runs.
  • Open-sourced generated kernels on GitHub.

Results

  • Overall: 3.09Ă— average speedup over PyTorch on 100 KernelBench level-1 kernels (vs. AI CUDA Engineer as secondary baseline, measured under identical hardware on 8Ă— A100, CUDA 12.4, DeepSeek-R1 as backbone LLM).
  • GEMM: 4.06Ă— average speedup over PyTorch; cuPilot generates Tensor Core invocations (14/16 kernels), memory padding (10/16), layout/thread-block swizzle (12/16), double buffering (8/16), multi-stage pipeline (2/16), asynchronous copy (5/16), PTX-level optimization (3/16). AI CUDA Engineer achieves 0/16 on most of these.
  • CONV: 1.18Ă— over PyTorch.
  • Activation/normalization: 4.16Ă—; pooling: 3.85Ă—; loss functions: 6.87Ă—.
  • Roofline ablation: roofline-guided prompting reduces latency by 44.2% on average over two epochs across four representative kernels (both DeepSeek-R1 and Gemini-2.5-Pro).
  • RAG initialization ablation: external strategy pool reduces latency by 54.1% after one epoch on four representative kernels.
  • Per-kernel runtime: under 3 hours per epoch.

Limitations

  • Evaluated only on KernelBench level-1 (100 kernels); coverage of more complex/hierarchical benchmarks is not demonstrated.
  • Runtime of up to 3 hours per kernel per epoch makes it impractical for rapid iteration or large kernel libraries.
  • Strategy pool quality depends on historical data accumulation; cold-start performance (no prior history) is not separately quantified.
  • LLM backbone (DeepSeek-R1, Gemini-2.5-Pro) costs and API dependencies are not analyzed; performance with weaker/open-source models is not reported.
  • AI CUDA Engineer comparison may be partially confounded by their reliance on cuBLAS wrappers ("illusory optimization"), which the authors note but do not fully control for across all kernel types.

Relevance to Harnesses / Meta-Harnesses

cuPilot is a concrete example of a multi-agent meta-harness where the orchestration layer (SCE Manager) operates over an abstract semantic representation (strategies) rather than raw artifacts (code), delegating specialized subtasks to subordinate agents with well-defined interfaces. The strategy-level crossover loop, RAG-augmented initialization, and roofline-guided prompting injection are all harness-level concerns—analogous to how a meta-harness might coordinate tool selection, prompt construction, and context retrieval across agents. The explicit decoupling of "what to do" (strategy space, managed by SCE Manager) from "how to do it" (kernel space, managed by Translator + Revisor) is a transferable architectural pattern for any harness managing complex, multi-step code generation or optimization pipelines. The paper also demonstrates that mismatched representations between harness-level abstractions and agent-level execution are a root cause of failure—a diagnostic framing applicable to meta-harness design broadly.