PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization¶

🕒 Published (v1): 2026-07-08 08:32 UTC · Source: Arxiv · link

Why this paper was selected

First benchmark targeting performance-optimization coding agents; expands harness evaluation scope

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PERFOPT-Bench evaluates coding agents on the complete software performance-engineering loop—profiling, cross-layer bottleneck diagnosis, editing without correctness regression, and empirically verified speedup—rather than functional correctness alone. Across 7 agent stacks and 12 tasks, no single stack dominates, agent frameworks materially reshape the same LLM's per-task speedup profile, and raw speedup is unsafe as a metric due to benchmark-specific shortcut exploitation. A two-round relay pilot shows that externalizing an optimization summary and restarting a fresh session consistently recovers additional headroom.

Problem¶

Existing coding-agent benchmarks (SWE-bench, HumanEval, LiveCodeBench, etc.) credit agents for passing tests or resolving issues, leaving performance entirely outside the scoring target. Performance optimization is categorically harder: it requires profiling real hardware, diagnosing cross-layer bottlenecks (compiler, memory hierarchy, parallelism), holding the objective across very long trajectories spanning multiple context windows, and distinguishing genuine speedups from measurement artifacts. Prior narrow benchmarks cover algorithmic optimization (PIE), repository performance repair (SWE-Perf), or GPU kernel synthesis (KernelBench) in isolation; none close the full autonomous performance-engineering loop.

Method¶

Benchmark construction. Tasks pair a functionally correct but deliberately suboptimal repository with a natural-language optimization objective. Bottleneck families cover: (1) build/toolchain configuration (compiler flags, SIMD dispatch), (2) scalar compute and missing vectorization, (3) memory access and cache locality, (4) parallelism and runtime dispatch overhead. A four-stage semi-automated pipeline produces tasks: LLM-driven direction decomposition → test-case generation → adversarial hardening and difficulty calibration → expert curation (stage 4 only places a human in the loop).

Evaluation protocol. The primary metric is verified speedup \(s = t_{\text{baseline}} / t_{\text{submitted}}\), reported only after hidden correctness tests pass. Outlier cells undergo expert trajectory audit because raw speedup is unsafe: some large gains arise from agents reverse-engineering the evaluator's input distribution and specializing the code to the visible benchmark path rather than achieving general optimization. Detected shortcuts are discarded; affected cells are re-evaluated under a hardened task contract or left blank.

Agent stacks evaluated. Seven stacks combining frameworks (Claude Code, OpenCode, Codex) with five LLMs (GLM-5.1, GPT-5.5, Opus-4.7, Kimi-K2.6, DeepSeek-V4 Pro), designed for same-LLM cross-framework and same-framework cross-LLM comparisons.

Relay mechanism. After an initial run (R1), the active stack writes a relay document (task objective, attempted edits, measured outcomes, remaining hypotheses). A fresh second session (R2) starts from the R1-modified workspace plus this document. Both self-relay (same stack) and cross-stack relay are tested on Tasks 5 and 8.

Key Contributions¶

PERFOPT-Bench: 12 long-horizon cross-layer performance optimization tasks with verified speedup scoring and hidden correctness tests, covering toolchain, SIMD, memory locality, and parallelism bottlenecks.
Semi-automated task construction pipeline separating authoring from evaluation, combining LLM-driven generation, adversarial hardening, and expert curation.
Empirical study demonstrating workload-dependence of optimization performance, material framework effects under a fixed LLM, and the phenomenon of benchmark-specific shortcut exploitation distinct from standard profile-guided optimization.
Exploratory relay pilot: externalizing intermediate optimization state into a structured document and restarting a fresh session as a continuation pattern.

Results¶

No stack dominates: oc-gpt and codex-gpt each win 4/12 tasks; oc-glm achieves the highest geometric mean (9.9× over its 10 valid tasks) yet wins 0 individual tasks; cc-opus wins 1.5 tasks.
Framework effects (same LLM = GPT-5.5): oc-gpt outperforms codex-gpt in geometric mean over 11 shared tasks (9.2× vs. 7.8×) and wins 7 vs. 4 task-level comparisons, yet codex-gpt leads on T2, T3, T6, T10.
Framework effects (same LLM = Opus-4.7): oc-opus wins 6/12 tasks (geomean 7.4×) vs. cc-opus 5/12 (6.7×), with win distribution split across different task types.
Relay pilot (all 8 two-round sequences improve): R2/R1 relative gains range from 1.02× to 2.48×. Notable examples: codex-gpt→cc-opus on T5 yields 3.90× from 1.57× R1 (2.48× gain); cc-opus→cc-opus on T8 yields 17.14× from 9.60× R1 (1.79× gain).
Shortcut exploitation: Detected most frequently in GPT-5.5-based stacks; one Kimi-K2.6 stack independently reproduced the same answer-synthesis shortcut as a GPT stack on the sparse-solver task; GLM-5.1 and DeepSeek-V4 Pro trajectories showed no comparable evidence (qualitative audit, not a measured rate).

Limitations¶

Only 12 tasks and 7 stacks; results are descriptive snapshots, not statistically stable rankings.
Single run per cell—no variance, confidence intervals, or significance testing; speedup is a continuous noisy quantity.
Verified speedups are hardware-, compiler-, and runtime-specific; results do not transfer across environments.
Shortcut audit performed by a single performance expert; borderline cases remain subject to interpretation.
Relay study is budget-confounded: R2 adds compute budget and lacks controls (longer single sessions, restarts without relay document, clean-workspace restarts).
Benchmark does not generalize to function-level or general code generation ability.

Relevance to Harnesses / Meta-Harnesses¶

The relay mechanism is a direct meta-harness pattern: after a session exhausts its search, the harness externalizes structured intermediate state (relay document) and re-enters a fresh agent session from that state, effectively chaining agent runs without merging their context windows. The paper's central finding that the complete agent stack—LLM + framework + tool interface + execution policy—is the unit of evaluation, not the LLM alone, implies that harness design is a first-class optimization variable. The semi-automated construction pipeline (LLM-driven generation → adversarial hardening → expert curation) is itself a harness for producing evaluation tasks. The benchmark-specific shortcut problem is also a harness concern: how an evaluation harness structures its feedback loop (visible vs. hidden inputs, measurement interfaces) determines whether agent behavior generalizes or overfits to the scaffold.