Skip to content

Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent

🕒 Published (v1): 2026-07-16 03:49 UTC · Source: Arxiv · link

Why this paper was selected

Trace-driven production benchmark + optimization agent for LLM-generated GPU kernels; practical infra-level eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Atrex-Bench is a GPU kernel-generation benchmark sourced from full-cluster production inference traces, scored with an importance-weighted, per-problem roofline metric. Evaluating six frontier LLM coding agents reveals the best achieves only \(S_{agg} = 0.107\) (10.7% of hardware roofline), and apparent correctness rates are inflated by PyTorch fallbacks rather than target-DSL kernels. The companion Atrex-Kernel-Agent (AKA) closes much of the gap via a profile-driven iterative measure–revise loop with optimization dropout and a layered knowledge base.

Problem

Existing GPU kernel-generation benchmarks (KernelBench, BackendBench, TritonBench, etc.) draw problems from synthetic or curated sources that diverge from deployed workloads on three axes: (1) they assume a uniform shape distribution rather than the production skew where the top-5 operators carry ~64% of GPU wall-time; (2) they apply unweighted task averages that treat rare elementwise ops equally with fused-attention paths; and (3) they score against mutable software baselines rather than per-shape hardware roofline ceilings. Consequently, existing benchmarks cannot predict deployability, and their scores do not reflect real-environment kernel-generation ability.

Method

Atrex-Bench constructs its problem set from online production inference traces on XPU-A3 and H20 clusters (>10k accelerators, ~20 deployed models, four serving frameworks: vLLM, SGLang, AITER, RTP-LLM). The pipeline: 1. Attaches a non-intrusive tracer combining Python-frame context with vendor runtime traces, correlated via IDs and timestamps. 2. Maps raw kernel records to computation-level operator families (30 operators, 440 hot shapes from 1,303 profiles). 3. Derives per-shape roofline bounds: \(T_{\text{roofline},j} = \max\!\left(\frac{F_j}{P_{\tau_j}},\, \frac{M_j}{\beta}\right)\), stored in hidden roofline.json. 4. Computes importance weights \(w_i\) as application-card-hour-weighted, phase-aware GPU-time shares (Eq. 3), normalized to sum to 1. 5. Scores via \(S_{agg} = \sum_i w_i S_i\), where per-operator achievement \(S_i\) is the median per-shape roofline ratio \(S_j = T_{\text{roofline},j}/T_{\text{cand},j}\) over correct shapes (or 0 if none pass). Provenance and roofline artifacts are hidden from candidate agents. 6. Evaluates through a three-stage gate: compile → correctness (5 random seeds, \(\text{atol}=10^{-2}\), \(\text{rtol}=5\times10^{-2}\)) → performance (frequency-locked, L2-flushed per iteration, subprocess-isolated).

Atrex-Kernel-Agent (AKA) pairs a profile-driven measure–revise search loop with optimization dropout—a partial restart that masks stale iteration memories while preserving the current best kernel and audit trail, allowing a fresh sub-agent to explore a different optimization direction. It uses a layered knowledge base comprising 298 reference-kernel files, 244 optimization-knowledge documents, and upstream open-source projects for API/ISA lookup.

Key Contributions

  • Atrex-Bench: first kernel-generation benchmark sourced from full-cluster production traces with importance-weighted, per-problem roofline scoring.
  • Release contract: packages production-derived references, hidden provenance (metadata.json), hidden roofline artifacts (roofline.json), and refreshable importance weights.
  • Empirical evaluation of six frontier coding agents quantifying the deployability gap (\(S_{agg}^{\text{best}} = 0.107\)) and exposing a correctness illusion via post-hoc target-DSL-dominance analysis (e.g., Qwen3.7-Max achieves 84.8% correctness but only 43.8% FlyDSL adoption).
  • AKA: profile-driven optimization agent that converts 0% FlyDSL fallbacks to near-100% FlyDSL kernels (6.7Ă— improvement on attention_forward) and raises roofline achievement from \(S = 0.28 \to 0.42\) on a stronger model, exceeding hand-tuned production baselines on both attention operators.

Results

  • Best vanilla model (across six frontier agents): \(S_{agg} = 0.107\); no agent matches the deployed production kernel.
  • Top-5 operators (unified_attention 36.1%, fused_moe 10.4%, block_scaled_mm 8.5%, fp8_blockscale_fused_moe 4.7%, paged_attention_decode 4.0%) carry ~64% of aggregate weight, making failures there extremely costly.
  • Qwen3.7-Max: 84.8% correctness pass rate but only 43.8% FlyDSL adoption—majority of "correct" outputs are PyTorch fallbacks, not target-DSL kernels.
  • Agents achieve several times more of the roofline on memory-bound shapes than compute-bound shapes (exact ratio not given in the excerpt).
  • AKA on weaker model: converts 0% FlyDSL → ~100% FlyDSL on attention_forward, yielding 6.7Ă— speedup over the fallback.
  • AKA on stronger model: \(S\) improves from 0.28 → 0.42, overtaking hand-tuned production baseline on both attention operators.

Limitations

  • Benchmark currently targets a single DSL (FlyDSL), which is absent from pre-training corpora by design; Triton, Gluon, and CuteDSL support is deferred to future releases.
  • Evaluation in §4 is conducted primarily on XPU-A (a desensitized non-NVIDIA accelerator name); H20 coverage is part of the design but results in the excerpt focus on one hardware target.
  • AKA results come from a "controlled case study" rather than a full-sweep benchmark evaluation, limiting generalizability claims.
  • The benchmark covers only 30 operators and 440 shapes from ~20 models; tail operators (20 of 30) collectively contribute only 20.4% of the weight, so those long-tail results carry limited signal.
  • Production weights require periodic recomputation from live traces; stale weights reduce production fidelity over time.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the question of whether LLM coding agents produce deployment-grade outputs—a core concern for any agentic system that generates executable artifacts. It establishes that correctness is an insufficient proxy for agent capability when agents can satisfy the correctness gate via fallback delegation rather than solving the target task, which is a broadly applicable failure mode for tool-using agents. The AKA agent demonstrates a concrete agentic loop pattern—iterative measure–revise with knowledge retrieval and optimization dropout to escape local optima—that is transferable to other domains requiring iterative optimization under measurable feedback. The benchmark infrastructure (hidden provenance, importance weighting, roofline ceiling) also provides a model for rigorous evaluation of agents on production-realistic task distributions rather than synthetic suites.