The Capability Frontier: Benchmarks Miss 82% of Model Performance¶

🕒 Published (v1): 2026-06-25 10:20 UTC · Source: Arxiv · link

Why this paper was selected

Benchmarks miss 82% of capability; challenges eval methodology for agent harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Standard LLM benchmarks report single-model, single-run accuracy and thereby systematically understate what is achievable with existing models. This paper introduces the Capability Frontier—a debiased Pareto frontier over quality and cost under optimal per-prompt model and generation selection—and shows it reduces error rates by 54% (model routing alone) or 82% (routing + posthoc multi-run selection) relative to the best single model, while matching SOTA accuracy at 85% lower cost.

Problem¶

Single-model, single-run benchmark evaluation produces two opposing distortions: (1) it underestimates achievable performance because different models specialize on different prompt subsets (complementarity is ignored), and (2) naive oracle estimators that take the max over sample means overestimate gains by up to 8.7% in accuracy and 88% in cost due to finite-sample positive selection bias (the "optimizer's curse"). No prior framework corrects for both effects simultaneously.

Method¶

The authors define the true oracle as $O_{\text{true}} = \frac{1}{N}\sum_n \max_l \mu_{nl}$, where $\mu_{nl} = \mathbb{E}[\phi_{nlg}]$ is the true expected performance of model $l$ on prompt $n$. Since $\mu_{nl}$ is unobservable, the biased estimator $O_{\text{biased}} = \frac{1}{N}\sum_n \max_l \bar{\phi}_{nl}$ systematically overshoots because the argmax preferentially picks positive outliers; bias decays as $O(G^{-\lambda})$ with $\lambda \in [0.25, 0.75]$.

Two debiasing methods are proposed: 1. Extrapolation: fit $O^{\text{biased}}(G) = \alpha + \beta G^{-\lambda}$ (or a smooth-transition variant for small $G$) and take $\hat{O}_{\text{true}} = \alpha$. 2. Probabilistic Graphical Model (PGM): a generative model with latent per-prompt difficulty $D_n \sim \text{Beta}$, topic assignment $T_n \sim \text{Categorical}$, and per-model-topic aptitude $A_{tl} \sim \text{Beta}$. A feedforward network serves as the link function $\pi_{nl} = f(D_n, A_{T_n,l})$; inference via stochastic variational inference with factorized posterior yields $\hat{\pi}_{nl}$, from which $O_{\text{true}} = \frac{1}{N}\sum_n \max_l \hat{\pi}_{nl}$.

The Capability Frontier is the Pareto surface swept by varying $\alpha \in [0,1]$ in the scalarized objective $\phi(\alpha) = \alpha Q^*_{nlg} + (1-\alpha)(-C^*_{nlg})$, with debiasing applied separately to quality and cost dimensions. A posthoc oracle additionally models selecting the best generation among $k$ samples per model after observing outputs, yielding $O^{k\text{-shot}}(k) = \frac{1}{N}\sum_n [1 - \prod_l (1-\pi_{nl})^k]$.

Evaluated on 21 LLMs $\times$ 16 benchmarks with $G=10$ generations per prompt-model pair. Controlled PGM simulations with $G=10{,}000$ sweep topic-distribution entropy to characterize how data heterogeneity drives oracle gains.

Key Contributions¶

Capability Frontier framework: rigorous quality-cost Pareto frontier characterizing achievable system-level performance under optimal routing and generation selection.
Bias characterization: formal proof that $O_{\text{biased}}$ overstates $O_{\text{true}}$; bias decays as $O(G^{-\lambda})$ ($\lambda \in [0.25, 0.75]$) in both Gaussian and Bernoulli cases.
Two debiasing methods: extrapolation-based correction and PGM-based posterior inference, both validated on synthetic data with known ground truth.
Empirical evaluation: 21 LLMs across 16 diverse benchmarks (coding, reasoning, medical, factuality, instruction-following, agentic); 54% error reduction from routing, 82% with posthoc selection ($k=10$), 85% cost reduction at matched accuracy.
Simulation study: synthetic PGM confirms oracle uplift increases monotonically with topic-distribution entropy, providing mechanistic grounding for why heterogeneous workloads benefit most from routing.

Results¶

Error rate reduction (routing only, $\alpha=1$): 53.7% average across 16 benchmarks vs. the per-benchmark SOTA single model (Table 1); e.g., GPQA Diamond 94.0% → 99.0% (82.7% error reduction), LeetCode Hard 79.1% → 87.7% (41.3% error reduction), Terminal-Bench 2.0 40.8% → 49.0% (13.9% error reduction).
Cost savings at matched SOTA quality: 85.2% average cost reduction (Table 2); e.g., Terminal-Bench 2.0 $2.6084 → $0.2539 (90.3%), TruthfulQA $0.0038 → $0.00003 (99.5%).
Posthoc oracle ($k=1$): 66% error reduction vs. SOTA; at $k=10$: 82% error reduction.
Naive oracle bias: average 1.2% quality overestimation, 37.5% cost overestimation; cost bias larger due to skewed cost distributions.
Model reliability: GPT-5-mini most reliable (90.2%), GLM-4.6 least reliable (76.3%); no significant correlation between reliability and quality or cost.
Simulation: oracle uplift near zero at low entropy (single-topic workloads), monotonically increasing to maximum at uniform topic mixture.

Limitations¶

$G=10$ generations per prompt is insufficient for the $O(G^{-\lambda})$ asymptotic regime ($G \geq 50$ needed); extrapolation carries uncertainty at small $G$.
Agentic benchmark gains are likely underestimated: routing is fixed per trajectory rather than per step; optimal per-step routing is combinatorially hard and not evaluated.
Posthoc oracle assumes a perfect, cost-free judge; real verifiers introduce errors and non-zero costs that reduce achievable gains.
PGM structural choices (factorized posterior, neural link function) are ad hoc and may not generalize across all workload types; conditional independence across generations may underestimate within-model correlations from shared training.
Study does not evaluate practical routing policies that can approach frontier performance under deployment constraints—only the theoretical upper bound is characterized.

Relevance to Agentic AI / LLM Agents¶

This paper directly challenges how multi-agent and agentic systems are benchmarked: if single-model evaluation misses 54–82% of achievable performance, then leaderboard comparisons between agentic frameworks may systematically misreport the state of the art. The posthoc oracle result is particularly relevant to agent settings where a judge/verifier (e.g., an execution environment) is available to score candidate outputs—sampling $k$ trajectories per agent and selecting the best is a standard practice in code generation and tool-use agents, and this work provides a principled upper-bound framework for quantifying how much is left on the table. The finding that gains scale with workload entropy has direct implications for designing agentic benchmarks: heterogeneous multi-domain task suites (like tool-use or long-horizon planning benchmarks) have the most to gain from model diversity, motivating mixture-of-agents and routing architectures over monolithic single-LLM agents.