When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models¶

🕒 Published (v1): 2026-06-25 17:06 UTC · Source: Arxiv · link

Why this paper was selected

Co-failure ceiling theorem fundamentally bounds routing/voting/MoA gain across 67 frontier models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Any selection policy over a pool of LLMs (routing, voting, cascades, mixture-of-agents) has accuracy bounded by $1 - \beta$, where $\beta$ is the rate at which all models fail on the same query simultaneously. The industry diagnostic—mean pairwise error correlation $\rho$—cannot identify $\beta$ and systematically underprices the co-failure tail, especially as pool size grows. Across 67 frontier models, a correctly calibrated Gaussian copula still underestimates the true co-failure tail by ~2.5×, and learned routers capture near-zero of the oracle gain.

Problem¶

Multi-model orchestration systems are evaluated and designed using mean pairwise error correlation $\rho$ as the key diversity signal. The paper shows $\rho$ is the wrong quantity: it cannot distinguish error distributions with identical marginals and identical pairwise correlations but different joint all-models-wrong rates $\beta$. Practitioners therefore have no principled upper bound on what routing, voting, or MoA can actually deliver before deploying expensive infrastructure.

Method¶

Theoretical: - Prove that for any selection policy whose output is almost surely one of the member models' answers, accuracy $\leq 1 - \beta$ (Prop. 1). The oracle gain localizes as $G = \Pr[\text{single-best wrong}] - \beta$, supported entirely on the resolvable mass where models disagree. - Derive a $\$0$ pre-deployment certificate: fit a Clopper–Pearson lower bound $\beta_{lo}(K, n, \delta)$ from $K$ all-wrong events over $n$ queries; if $(1 - \beta_{lo}) - a_{sb} <$ orchestration overhead, no selection policy in the class can pay for itself. - Prove (Prop. 2) that a single-factor Gaussian copula calibrated to $\bar{\rho}$ has zero lower tail dependence, so $\beta_{sf}(m) \to 0$ while true $\beta(m) \to \pi > 0$ under a common-shock mixture, making the underpricing ratio $\beta(m)/\beta_{sf}(m) \to \infty$ as pool size $m$ grows. - Show (Prop. 3) that pairwise $\rho$ cannot identify $\beta$ for $m \geq 3$.

Empirical: - Run pre-registered experiments on 15 models (9 provider families) across GSM8K, MMLU, ARC-Challenge, MATH-500, MMLU-Pro (100–200 queries/dataset, ~$47). - Expand to 67 models across 21 families (live OpenRouter catalog) on hard benchmarks: MATH-500, MATH-Hard Level-5, AIME 2024/2025, GPQA-Diamond, and execution-graded competitive programming. - Calibrate the Gaussian copula using tetrachoric (latent) correlation rather than Pearson-of-indicators—a century-old psychometric correction the LLM evaluation literature routinely omits. - Fit the full 67×67 pairwise-tetrachoric correlation matrix, project to nearest PSD, and Monte Carlo integrate the all-wrong event probability ($4 \times 10^5$ draws) to test whether residual co-failure excess is single-factor misspecification or a structural common-mode atom. - Train four router types (TF-IDF logistic, gradient-boosted correctness predictor, multiclass best-model predictor, GPT-5-mini LLM-as-router) and measure realized gain against oracle.

Key Contributions¶

Orchestration ceiling (Prop. 1): Accuracy $\leq 1-\beta$ for any selection policy; gain localizes to resolvable mass; Clopper–Pearson bound converts one held-out query set into a $\$0$ pre-deployment certificate.
$\rho$ cannot identify $\beta$ (Props. 2–3): Pairwise correlation is sufficient for bivariate error laws but discards higher-order tail dependence; the underpricing ratio diverges in pool size $m$, forced by any positive tail dependence.
Market-scale measurement (67 models, 21 families): Empirical $\beta = 0.052$ on MATH-500 vs. correctly tetrachoric-calibrated single-factor prediction $\beta_{sf} = 0.021$ (~2.5×, 90% CI 1.7–3.4×); the full 67×67 pairwise Gaussian copula predicts only $\beta_{full-\Sigma} = 0.023$, leaving a 2.25× residual—the signature of a common-mode atom no pairwise calibration can represent.
Two regimes identified: Ceiling-bound (open-ended math, code: $\beta > 0$, tail structurally underpriced) vs. realizability-bound (science/GPQA multiple-choice: $\beta \approx 0$, ceiling open but routing gain unrealizable due to insufficient query-level signal).
Calibration trap diagnosed: Naive Pearson-of-indicators calibration produces spurious 30× underpricing artifact; tetrachoric calibration reduces this to ~2.5×.

Results¶

Oracle gain: $G = 0.044$ [0.027, 0.062] on saturated mix; $G = 0.120$ [0.075, 0.155] on hard (MMLU-Pro); both CIs exclude zero (Table 1).
Realizable router gain: Near zero across all four router types. TF-IDF logistic: 0.906 vs. single-best 0.901 (9% of $G$, CI spans zero). Gradient-boosted correctness predictor: $-0.09$ of $G$. Multiclass best-model predictor: $-1.27$ of $G$ (actively harmful). LLM-as-router (GPT-5-mini): routes to single-best on 100% of queries, captures exactly 0 of $G$.
Co-failure tail (15-model pool): $\beta = 0.033$ [0.019, 0.054] on saturated mix; naive Pearson single-factor predicts $\beta_{sf} = 0.0011$ (30× underpricing); tetrachoric residual is single-digit.
Market scale (67 models, MATH-500): $\beta = 0.052$ (k=17 all-wrong events over n=330); tetrachoric $\bar{\rho} = 0.78$ predicts $\beta_{sf} = 0.021$; residual ≈ 2.5× (90% CI 1.7–3.4×); full-$\Sigma$ Gaussian copula predicts 0.023, leaving 2.25× beyond any Gaussian.
Pool size effect: Tetrachoric underpricing ratio rises monotonically from 1.0 at $k=2$ to median 2.5 (5–95% band [2.1, 2.7]) at $k=67$ over random composition resampling (60 per $k$).
Code domain replication: $\beta = 0.079$ (k=5, 63 problems); tetrachoric single-factor underprices by 3.1×; full-$\Sigma$ Gaussian copula leaves 1.7× residual (bootstrap 90% CI [1.5, 6.2] excluding 1).
GPQA-Diamond multiple-choice: $\beta \approx 0$ (co-failure tail vanishes); re-asking in free-response format reopens it to $\beta = 0.127$ (five-judge LLM panel, $\kappa$ 0.73–0.92), localizing the effect to task format not subject matter.
Diversity vs. same-model MoA: Diverse low-$\rho$ ensemble beats high-$\rho$ Self-MoA at matched quality.

Limitations¶

All-wrong counts are small (k=17 on MATH-500, k=5 on code, k=6 on MMLU-Pro), producing wide Clopper–Pearson intervals; the 2.5× point estimate carries real uncertainty.
Code replication uses only 18 models (not 67) with a strict-but-unofficial judge.
Market-scale routing statement (67 models) rests on the Prop. 1 certificate, not an end-to-end routing run, because per-query prompts were not logged for the expanded pool.
The two open-ended math benchmarks (MATH-500, MATH-Hard) are the same task family at different difficulties—not independent domain replications.
AIME 2024/2025 results are potentially contaminated for older models whose training postdates the release; the paper notes this.
The common-mode atom is identified structurally (residual beyond any exchangeable pairwise-calibrated copula) but not characterized—its source (shared training data, benchmark contamination, architectural monoculture) is not pinned down.
Router experiments are limited to the 15-model pool where prompts were logged; stronger architectures (e.g., fine-tuned classifiers) are not tested.

Relevance to Agentic AI / LLM Agents¶

This paper directly bounds the theoretical headroom for multi-agent LLM systems: any architecture where the final answer is selected from member outputs—including MoA pipelines, debate/self-consistency, and agentic routing layers—cannot exceed $1 - \beta$. The empirical finding that $\beta$ is structurally underestimated by pairwise diversity metrics is critical for practitioners designing agent ensembles, because it means the apparent diversity of a large heterogeneous pool does not translate into orchestration gain on hard, open-ended tasks. The task-format result ($\beta$ vanishes on multiple-choice but resurges on free-response) is particularly relevant as agentic systems shift toward open-ended generation; the $\$0$ certificate offers a practical pre-deployment tool for deciding whether an agentic orchestration layer can pay for its inference cost on a given task distribution.