Skip to content

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

🕒 Published (v1): 2026-05-16 05:25 UTC · Source: Arxiv · link

Why this paper was selected

AgentKernelArena: generalization-aware benchmark for GPU kernel optimization agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentKernelArena is a 196-task benchmark for evaluating AI coding agents on GPU kernel optimization, covering HIP-to-HIP, Triton-to-Triton, and PyTorch-to-HIP tasks. Unlike prior benchmarks that score single LLM calls, it evaluates complete multi-turn agentic workflows in isolated workspaces with a gated compile→correctness→performance pipeline. A novel unseen-configuration generalization protocol reveals that agents frequently hardcode shape-specific assumptions that break on inputs never seen during optimization.

Problem

Existing kernel benchmarks (KernelBench, TritonBench, robust-kbench) evaluate isolated single LLM outputs or light iterative prompting — they do not test full agent loops with tool use (compiler invocation, profiling, iterative refinement). None include kernel-to-kernel optimization tasks (only specification-to-kernel), and none test whether reported speedups transfer to input configurations the agent never observed. This makes cross-agent comparison unreliable and hides systematic overfitting to visible test shapes.

Method

Each of 196 tasks gives an agent a timestamped, isolated workspace containing source files, build infrastructure, and evaluation scripts. Agents receive a prompt specifying task type, target functions, shell commands for compile/correctness/performance, optional hardware cheatsheets (GPU arch guide, HIP/Triton best-practices), and an instruction to produce up to max_iterations=3 successive kernel versions. The agent runs with full shell access for up to 3600 s; after termination a centralized evaluator independently applies a gated pipeline: compilation (hipcc/AST validation) → correctness (output matching reference across all visible shapes) → performance (10 warmup + 100 timed iterations via torch.cuda.Event).

Scoring combines all three axes: $\(\text{Score}(k) = 20 \cdot \mathbf{1}[\text{compiles}] + 100 \cdot \mathbf{1}[\text{correct}] + 100 \cdot s_k \cdot \mathbf{1}[\text{correct}]\)$ where \(s_k = t_\text{base}/t_\text{opt}\). The linear performance term avoids saturation; any correct kernel strictly dominates any incorrect one regardless of speedup.

The unseen-configuration generalization protocol uses Cursor Agent / claude-opus-4-6-high to generate 8 structurally diverse unseen configurations per task across six categories (edge-case/boundary, scale-up \(\geq 2\)–\(4\times\), scale-down \(\leq 2\)–\(4\times\), alignment-stress with prime or non-power-of-two sizes, asymmetric aspect ratio, production-realistic). The generalization gap is: $\(\Delta_g = \frac{\bar{s}_\text{seen} - \bar{s}_\text{unseen}}{\bar{s}_\text{seen}}\)$ and the key correctness metric is conditional correctness \(P(\text{opt correct} \mid \text{orig correct})\).

Tasks: HIP-to-HIP (24, from GPU Mode community), Triton-to-Triton (148: 118 from vLLM, 30 from ROCmBench), PyTorch-to-HIP (24). Experiments run on AMD Instinct MI300X / ROCm 7.1.1 with three agents: Cursor Agent, Claude Code, Codex Agent.

Key Contributions

  • Agent-centric benchmark with 196 tasks that evaluate full multi-turn tool-using workflows, not isolated LLM outputs.
  • Gated evaluation pipeline (compile → correctness → performance) executed by a centralized evaluator separate from the agent workspace, ensuring reproducibility and fair cross-agent comparison.
  • First unseen-configuration generalization protocol for agentic GPU kernel optimization, revealing systematic shape-overfitting.
  • Modular extensible framework: new agents, tasks, and hardware targets are added via configuration files.
  • Multi-shape visible evaluation: agents see diverse input shapes during optimization to discourage single-shape tuning, separate from the held-out unseen set.

Results

  • Near-perfect compilation rates across all categories; notable exception: Cursor / GPT-5.4 High on PyTorch-to-HIP drops to 69.4%.
  • HIP-to-HIP (24 tasks): best mean speedup 6.69× (Claude Code / Opus 4.6); geometric mean 1.33–3.31×.
  • Triton-to-Triton (148 tasks): best mean speedup 2.13× (Cursor / Opus 4.7 High); geometric mean 1.01–1.31×; fast2 <11% for all configurations, reflecting Triton's compiler-managed optimization headroom.
  • PyTorch-to-HIP (24 tasks): best mean speedup 6.89× (Cursor / Opus 4.6 High); geometric mean 2.19–4.64×; fast2 58–83%.
  • Generalization — HIP-to-HIP: conditional correctness 93.6%–100%; most configurations gain speedup on unseen inputs (e.g., Cursor / Opus 4.7 High +23%).
  • Generalization — Triton-to-Triton: conditional correctness 91%–95%; speedups transfer roughly on-diagonal.
  • Generalization — PyTorch-to-HIP: conditional correctness drops to 60%–90%, indicating widespread shape-specific hardcoding in from-scratch kernel generation.

Limitations

  • Hardware scope limited to AMD Instinct MI300X; generalization to NVIDIA or other backends is untested.
  • Unseen configurations are not publicly released (only generation script), making external replication of the generalization results indirect.
  • The max_iterations=3 directive is a natural-language instruction rather than a hard tool-call cap, so agents may vary substantially in actual iteration depth.
  • Triton-to-Triton tasks provide limited optimization signal (\(\leq 2.13\times\) mean speedup) because Triton's compiler already handles many low-level optimizations, potentially under-differentiating agents.
  • Only three production agents evaluated; open-source or custom agentic systems require community integration effort.

Relevance to Harnesses / Meta-Harnesses

AgentKernelArena is itself a meta-harness: it wraps heterogeneous production agents (Claude Code, Cursor, Codex) behind a uniform workspace-isolation and gated-evaluation shell, separating the agent loop from the scoring infrastructure — the canonical meta-harness pattern of "agent runs freely; evaluator runs independently." The centralized evaluator that fires after agent termination mirrors how scoring harnesses decouple generation from assessment in multi-agent pipelines. The unseen-configuration generalization protocol is a harness-level design pattern for detecting overfitting in iterative agentic systems, directly applicable to any domain where agents refine outputs against visible test cases. The modular "add agent/task/hardware via config" architecture demonstrates how meta-harnesses can be built for extensibility rather than baked around specific agent APIs.