Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision¶

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Under unreliable supervision (weak LM supervisors or time-constrained humans), DPO fails to improve beyond the SFT baseline due to a fundamental tension between preventing overoptimization and enabling large model updates. The authors propose Iterative Label Refinement (ILR), which redirects comparison feedback toward improving SFT training data rather than updating model weights, achieving consistent gains on math, code, and instruction-following tasks where DPO stagnates.

Problem¶

Standard post-training pipelines (SFT → RLHF/DPO) assume reliable supervision. As tasks grow harder, human demonstrations and preference comparisons become noisy. DPO overoptimizes on unreliable preference signal and requires heavy KL regularization (large β) to prevent it, but strong regularization prevents the large model updates needed to correct errors inherited from the unreliable SFT stage — a dilemma with no good resolution within DPO's framework.

Method¶

Iterative Label Refinement (ILR) replaces RLHF-style model updates with iterative SFT dataset updates: 1. Split the SFT dataset into two halves; train separate models on each half. 2. Each model cross-labels the half it was not trained on (avoiding memorization artifacts). 3. An unreliable annotator (small LM or time-constrained human) compares the model-generated proposal against the original demonstration; if the proposal is preferred, it replaces the original. 4. A new SFT model is trained from scratch on the refined dataset. 5. Repeat for K rounds, updating at most α = 0.15 fraction of labels per round (sorted by annotator confidence).

Re-training from scratch each round avoids overoptimization; the data-update mechanism exploits weak-to-strong generalization (the SFT model systematically outperforms its unreliable training data on held-out prompts).

Key Contributions¶

Empirical diagnosis of DPO failure under unreliable supervision: overoptimization requires large β, but large β prevents correcting SFT errors — a fundamental trade-off with no good operating point.
ILR algorithm: redirects comparison feedback to data improvement rather than model weight updates, bypassing the regularization dilemma.
Demonstration that comparison feedback (easier to elicit than demonstrations) suffices to improve SFT data quality iteratively.
Human study confirming results hold with realistic, time-constrained annotator noise, not just LM-simulated noise.

Results¶

GSM8K (2B→7B): SFT+ILR improves accuracy across 6 rounds; SFT+DPO stays flat or declines.
GSM8K (7B→70B): ILR shows larger gains at scale (~0.67→0.72 accuracy range), suggesting ILR benefits from model scaling.
BIRD SQL (2B→70B): ILR consistently outperforms DPO over 6 rounds.
SaferPaca (2B→7B): ILR improves win rate; DPO plateaus.
Human study (Alpaca/AlpacaEval, Llama 3 70B): ILR outperforms DPO at all tested β values (0.01, 0.1, 0.5, 1.0) after 2 rounds; DPO with small β improves fast then plateaus, large β improves slowly.
With oracle comparison feedback, ILR can raise SFT label accuracy toward (and sometimes above) the accuracy of a model trained on ground-truth demonstrations.

Limitations¶

Experiments focus on DPO; findings may not fully generalize to PPO-based RLHF pipelines with explicit reward models.
Human study uses time-pressure as a proxy for capability constraints; more authentic weak supervision (e.g., competition math or hard coding tasks) may exhibit different dynamics.
ILR's benefit degrades over rounds as model outputs surpass human-level complexity, making comparisons harder for time-constrained annotators.
Naive replacement (without comparison feedback) causes model collapse, so ILR depends on availability of at least some comparison signal.

Relevance to Vision-Language Models¶

VLMs follow the same SFT → RLHF/DPO post-training paradigm as LMs, and supervision reliability is a sharper concern: visual content is harder for human annotators to evaluate correctly, especially for spatial reasoning, OCR, or fine-grained visual descriptions. ILR's core insight — that comparison feedback is easier to collect reliably than demonstrations, and should be used to curate data rather than tune weights — transfers directly to VLM post-training pipelines. The method is also relevant to RLAIF or model-based preference data pipelines for VLMs, where weak vision encoders or small VLMs play the role of the unreliable supervisor here.