Skip to content

Rethinking Code Performance Benchmarks for LLMs

🕒 Published (v1): 2026-07-08 16:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing function-level Python performance benchmarks for LLMs (EffiBench, Enamel, EvalPerf, Mercury) are largely unreliable: under 30-run repeated execution with Mann-Whitney U statistical testing, only 6.11% of benchmark-provided "performant" implementations are measurably faster than their canonical counterparts. The authors diagnose two root causes—measurement noise and insufficiently stressful test inputs—then propose a multi-agent generate-diagnose-repair harness that uncovers statistically significant improvements in ~24–25% of previously indistinguishable tasks.

Problem

Current performance benchmarks evaluate LLM-generated code efficiency using single or very few execution runs (1–6) without statistical significance testing, and inherit test suites designed for functional correctness rather than runtime stress. As a result, 93.89% of benchmark-provided "performant" implementations are statistically indistinguishable from their canonical baselines (\(p \geq 0.05\)), making it impossible to determine whether LLM-generated code is genuinely inefficient or whether the benchmark itself cannot expose real differences.

Method

Rigorous re-evaluation: Each of 1,538 tasks across four benchmarks is executed 30 times per implementation. Per-task significance uses a one-sided Mann-Whitney U test (\(p < 0.05\)); effect size uses Cliff's delta \(\delta\). Benchmark-level aggregation uses Wilcoxon signed-rank test on paired per-task mean runtimes.

Root-cause analysis: 308 non-significant tasks are manually open-coded (Cohen's Kappa = 0.93) to distinguish (a) no meaningful code change vs. (b) genuine algorithmic change masked by weak inputs. LLM-as-a-Judge (DeepSeek-V3.1, GPT-4o) extends labels to the remaining 1,136 tasks (human–LLM Kappa ≈ 0.72–0.75).

Multi-agent test-generation framework: A batch-based pipeline with three specialized agents: 1. Generator: produces performance-oriented test inputs targeting performance-critical code paths. 2. Diagnostician: identifies why generated tests fail to expose differences or violate correctness. 3. Repairer: fixes failing or non-discriminating tests.

The loop iterates until tests are deterministic, functionally correct, and stress the relevant algorithmic path. Validated suites are then used to re-run the Mann-Whitney comparison.

Key Contributions

  • Rigorous 30-run re-evaluation of 1,538 tasks across four benchmarks with statistical testing and effect-size analysis, revealing the near-universal failure of existing evaluation protocols.
  • Manual open-coding of 308 non-significant tasks distinguishing vacuous changes (99 tasks) from genuine optimizations masked by weak inputs (209 tasks), validated at scale via LLM-as-a-Judge.
  • A three-agent generate-diagnose-repair framework for performance-oriented test generation that preserves functional correctness while stressing performance-critical paths.
  • Demonstration that stronger test suites substantially change benchmark conclusions: 24.01%–25.43% of previously non-significant tasks become significant.
  • Design recommendations for future benchmarks: repeated execution + statistical testing, problem selection with genuine optimization headroom, and extension beyond function-level to class- or repository-level tasks.

Results

  • Baseline reliability: Only 94/1,538 (6.11%) performant implementations are significantly faster than canonical solutions under 30-run execution + Mann-Whitney U test; 93.89% are non-significant.
  • Per-benchmark breakdown: EffiBench 5.6% significant, Enamel 6.71%, Mercury 4.3%, EvalPerf 13.56% (highest, due to deliberate input design).
  • Effect sizes among significant tasks: 92/94 (5.98%) show large Cliff's delta (\(\delta \geq 0.474\)); mean runtime improvements range +6.6% to +27.1%.
  • Root-cause: Of 308 sampled non-significant tasks, 99 contain no meaningful performance change; 209 contain genuine algorithmic improvements obscured by weak inputs. At scale, LLMs label 73.94%–76.06% of remaining non-significant tasks as having potential performance impact.
  • Multi-agent framework impact: Among 1,345 previously non-significant tasks, DeepSeek-v3.1-generated tests reveal significant improvement in 323 tasks (24.01%); GPT-4o in 342 tasks (25.43%)—outperforming prior SOTA LLM-based performance test generation.
  • LLM re-evaluation with stronger tests: Re-evaluating GPT-4o-mini, Claude-Sonnet-4.5, and Gemini-2.5-Flash with the new tests reveals statistically significant performance improvements in 22.19% of evaluated cases.

Limitations

  • All experiments run on a single desktop (Intel i5-8500); results may differ under server-class hardware or containerized environments with different scheduling noise profiles.
  • The 30-run threshold, while standard in performance engineering literature, may still be insufficient for extremely noisy or hardware-sensitive workloads.
  • The multi-agent framework was tested with only two LLMs (DeepSeek-V3.1, GPT-4o); generalization to other models or non-Python languages is unverified.
  • Manual analysis covers only 308 of 1,444 non-significant tasks (a statistically valid sample, but not exhaustive); LLM-as-a-Judge agreement is moderate (\(\kappa \approx 0.72\)–\(0.75\)), not high.
  • Framework evaluation focuses on whether tests expose differences, not on false-positive rate (tests that stress wrong paths or inflate noise).
  • Study is confined to function-level Python tasks; class- and repository-level benchmarks are discussed but not evaluated.

Relevance to Harnesses / Meta-Harnesses

The paper's core artifact is a meta-harness for automated test-suite improvement: a three-agent pipeline (generate → diagnose → repair) that wraps an existing benchmark harness and iteratively improves its evaluation quality. This is a direct instance of a meta-harness pattern—a system that acts on the evaluation infrastructure itself rather than on the target code under test. The work also surfaces a general meta-harness design principle: benchmark harnesses must include statistically rigorous execution loops (repeated runs + hypothesis testing) as a first-class component, not an afterthought. For researchers building or evaluating agentic coding harnesses, this paper provides empirical grounding that naive single-pass execution measurement is insufficient and demonstrates how a multi-agent repair loop can raise test discriminability from 6% to ~24% without human intervention.