Towards Scalable Oversight via Partitioned Human Supervision¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Masashi Sugiyama (RIKEN); partitioned human supervision to scale oversight beyond expert level

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a scalable oversight framework that uses complementary labels — expert signals indicating which answer option is wrong rather than which is correct — to evaluate and train AI systems on tasks where no single human can provide ground truth. It derives an unbiased estimator of top-1 accuracy from these weak labels, proves finite-sample deviation bounds, and demonstrates that the estimator can substitute for ordinary accuracy in agentic training pipelines.

Problem¶

As AI systems approach or exceed expert human performance on cross-domain tasks, traditional supervision pipelines (SFT, RLHF, RLVR) break down because no single human expert can verify correctness. Existing scalable oversight approaches (weak-to-strong generalization, debate, recursive decomposition) still require either correct labels or decomposable tasks. This work addresses the orthogonal regime where experts can only certify that a given option is wrong, not identify the correct answer.

Method¶

The framework operates on multiple-choice tasks with $K$ options. Each question is routed to a domain specialist responsible for one class $k$; the specialist answers "Is the answer class $k$?" yielding either an ordinary label (yes) or a complementary label (no — the answer is definitely not $k$). Under the uniform wrong-index design, the complementary label $\bar{Y}$ is sampled uniformly from the $K-1$ non-correct classes.

From $n_c$ complementary labels with "weakly-correct" rate $\hat{q} = S_c / n_c$ (model prediction $\neq$ complementary label), the unbiased estimator is: $$\hat{A}_\text{comp} = (K-1)\hat{q} - (K-2)$$

Its variance is $(K-1)^2 q(1-q)/n_c$, exceeding the ordinary-label variance by factor $\frac{(A+K-2)n_o}{A \cdot n_c}$; matching variance requires $n_c = (1 + (K-2)/A)\,n_o$ complementary labels.

Two mixture estimators combine $n_o$ ordinary and $n_c$ complementary labels: 1. IVW: inverse-variance-weighted combination $\hat{A}_\text{IVW} = \hat{w}\hat{A}_\text{ord} + (1-\hat{w})\hat{A}_\text{comp}$ with plug-in optimal weight $\hat{w}$. 2. ML: closed-form maximum-likelihood estimator solving a quadratic score equation over the joint binomial likelihood.

Finite-sample Hoeffding and empirical Bernstein bounds are proved for both the complementary-only and mixture estimators.

For agentic training, the complementary-label estimator replaces ordinary accuracy as the fitness signal inside agent search pipelines (ADAS and AFlow), enabling automated agent improvement without ground-truth labels.

Key Contributions¶

Scalable oversight protocol: routes questions to partitioned domain experts who supply complementary labels; formally reduces to a uniform wrong-index design provably satisfying unbiasedness.
Unbiased complementary-label estimator (Corollary 1) with closed-form variance and sample-size requirements to match ordinary-label variance (Eq. 5).
Two mixture estimators (IVW and ML) combining scarce ordinary and abundant complementary labels, both with practical plug-in forms.
Finite-sample deviation guarantees (Theorem 2 for $\hat{A}_\text{comp}$; Proposition 3 and Theorem 4 for $\hat{A}_\text{mix}$) via Hoeffding and Bernstein inequalities.
Agentic training demonstration: plugs the complementary-label estimator as a fitness function into ADAS and AFlow agent search, showing performance improvement without ground-truth labels.

Results¶

Statistical validation (MMLU-Pro, MedQA-USMLE, GPQA, MATH) with GPT-5-nano/GPT-4.1-nano: IVW and ML consistently achieve within-run standard deviations narrower than ordinary-only or complementary-only baselines; mean accuracies closely track the oracle Ord-Eval reference (e.g., IVW: 77.97 ± 1.58 vs. Ord-Eval 77.97 on MMLU-Pro; 91.86 ± 1.11 vs. 92.66 on MedQA-USMLE).
Comp-no (same sample size, no correction) has wide variance (e.g., ±12.49 on MMLU-Pro) making it unreliable despite occasionally close point estimates.
Real-world evaluation on EDINET-Bench (Japanese finance) and Medical Abstracts (English clinical text): IVW achieves lowest within-run variance on Medical Abstracts (±0.73 vs. ±0.96 for Ord, ±1.13 for Comp); ML achieves lowest variance on EDINET Extended (±2.26).
Agentic training: replacing ordinary accuracy with the complementary-label estimator in ADAS/AFlow pipelines improves downstream agent performance on tasks where ground truth is unavailable.

Limitations¶

Requires the uniform wrong-index assumption (Eq. 1): each expert is queried for a uniformly random class; systematic bias in class assignment violates the unbiasedness proof.
Complementary labels are inherently weaker: matching ordinary-label variance requires up to $(K-1)/A$ times more labels, which may be costly when $A$ is small or $K$ is large.
The ML estimator requires a reliable pilot estimate of $A$; it is less robust than IVW in low-data regimes.
The mixture bounds under data-dependent weights (plug-in $\hat{w}_\text{IVW}$) require sample splitting or union-bound corrections, loosening the constants.
Empirical validation of agentic training is a proof-of-concept and does not yet demonstrate scaling to frontier superhuman task difficulty.

Relevance to Agentic AI / LLM Agents¶

This work directly enables evaluation and self-improvement of LLM agents on tasks beyond human ability — the core bottleneck in the superalignment regime. By replacing oracle accuracy with an unbiased estimator derived from partitioned expert feedback, it unlocks reward signals for agent search pipelines (ADAS, AFlow) in domains where traditional verifiers cannot be constructed. It complements reward-model and RLHF approaches by providing a statistically grounded alternative signal when no single annotator can certify correctness, and its finite-sample guarantees make it suitable as a principled drop-in fitness function for automated agent design.