Skip to content

Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration

🕒 Published (v1): 2026-07-20 04:09 UTC · Source: Arxiv · link

Why this paper was selected

Fixed-pool diagnostic separates oracle gap from signal fidelity; clarifies when verifier pipelines help

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Test-time collaboration (self-consistency, best-of-N, verifier pipelines) does not uniformly improve LLM reasoning; gains are bounded first by the oracle gap (recoverable mass in the candidate pool) and then by signal fidelity of the selector. This paper introduces OracleGap, a fixed-pool diagnostic that decomposes selector net gain into recoverable mass, signal coverage, conditional quality, and harm to already-correct outputs, yielding a pre-deployment decision workflow.

Problem

Test-time collaboration methods are routinely credited with broad reasoning gains, yet empirical results are uneven and sometimes negative. No principled, measurement-level framework existed to explain when a training-free selector or verifier should be expected to help, or to attribute gains to specific, auditable properties of the candidate pool and selection signal rather than to agent topology or count.

Method

For each input \(x\), a fixed candidate pool \(C(x) = \{y_1, \ldots, y_k\}\) with \(k=5\) is generated. The framework defines: - Recoverable mass: \(P(\text{recoverable})\), where \(\text{recoverable}(x) = \text{ref\_wrong}(x) \wedge \text{any@}k\text{\_correct}(x)\) - Oracle gap: score of any-of-\(k\) oracle minus the reference, an upper bound on all fixed-pool selectors

Net gain is decomposed as: $\(\text{gain} = P(R \wedge D)\,q - P(C \wedge D)\,h\)$ where \(R\) = recoverable, \(C\) = reference correct, \(D\) = signal defined, \(q\) = conditional selection quality on recoverable+defined examples, \(h\) = harm rate on reference-correct+defined examples.

Verifier fidelity is measured independently as candidate-level agreement (accuracy, FPR, FNR, MCC) between verifier verdicts and official labels. A selector ladder (sample0, L1 same-family LLM, L3 cross-family LLM, L4-gen generated tests, L4-public public tests, oracle) is evaluated on LiveCodeBench (1,055 tasks, \(k=5\), 3 seeds), MATH Level-5 hard, GPQA-Diamond, and HumanEval+ using Qwen3.6-35B-A3B-BF16 as the main model.

Key Contributions

  • A training-free fixed-pool decomposition of selector gain into recoverable mass, signal coverage, conditional quality, and harm—making gains measurable without attributing them to topology
  • A three-seed LiveCodeBench selector ladder with hierarchical bootstrap confidence intervals on a unified common task set
  • Direct empirical measurement of verifier fidelity (MCC) as the bottleneck explaining the gap between public-test and generated-test selectors
  • Cross-task boundary evidence on MATH (symbolic equivalence beats self-consistency; LLM selectors are negative) and GPQA-Diamond (oracle gap 3.03%, 87.54% answer-identical pools—harm exceeds recoverable value)
  • A pre-deployment diagnostic workflow: estimate oracle gap → audit coverage/fidelity → compare capture vs. harm → deploy only if net-positive
  • A routing-\(\eta\) extension (supplementary) showing worker routing and candidate selection obey different constraints and must not be conflated

Results

  • LiveCodeBench: L4-public (MCC 0.825) gains +8.14 pp over first-sample baseline with zero harm; L4-gen (MCC 0.248) gains +2.70 pp (not statistically distinguishable from L3 LLM selector, 95% CI overlap); L1 same-family LLM gains +3.50 pp but harms 4.69% of correct outputs; oracle any@5 is +11.74 pp
  • LiveCodeBench fidelity: public-test verifier 92.98% candidate-level accuracy, MCC 0.825; generated-test verifier 53.05%, MCC 0.248; generated tests active on only 35.77% of rows
  • MATH Level-5: symbolic answer-equivalence selector beats self-consistency by +4.67 pp (95% CI [2.93, 6.40]); LLM selectors are negative (L1: −3.20 pp, L3: −1.87 pp); equivalence key available on 69.07% of rows
  • GPQA-Diamond: oracle gap is 3.03 pp; 87.54% of \(k=5\) pools are answer-identical; L1 nets −1.68 pp (fixes 5, harms 15); a weaker Qwen3.5-9B pool reduces recoverable mass to 0.67% with 94.44% identical pools

Limitations

  • Framework is restricted to fixed-pool selection; repair mechanisms that generate new candidates outside the pool are out of scope
  • Conclusions are derived from a single primary model (Qwen3.6-35B); robustness analyses use secondary models but are not the main ladder
  • Generated-test fidelity is measured with official labels, which are unavailable at deployment; the framework identifies the bottleneck but does not solve label-free fidelity estimation
  • GPQA-Diamond has no natural executable verifier, so the benchmark serves only as a boundary case rather than providing a full ladder comparison
  • The supplementary routing analysis is not formally validated as part of the main selector ladder

Relevance to Agentic AI / LLM Agents

This work directly challenges the default assumption in agentic system design that adding verification layers, critic agents, or multi-step selection reliably improves outputs—showing the gains are bounded by properties of the candidate pool that must be measured, not assumed. The oracle gap framework provides a concrete pre-deployment audit that agent pipeline designers can apply before investing compute in verifier or critic components. The distinction between selector critics, verdict critics, and repair critics maps cleanly onto common agentic roles (judge, ranker, reviser), and the harm decomposition is directly applicable to agentic loops where a critic can degrade a correct solution. The finding that signal fidelity (MCC) is the primary empirical bottleneck—not topology—reframes multi-agent collaboration as a data quality problem rather than an architectural one.