Skip to content

DOPD: Dual On-policy Distillation

🕒 Published (v1): 2026-06-29 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DOPD introduces an advantage-aware dual on-policy distillation paradigm that dynamically routes token-level supervision between a privileged teacher and a privileged student policy to address "privilege illusion" — the failure mode where gains from privileged inputs (e.g., visual annotations in VLMs) stem from information asymmetry rather than transferable capability. By partitioning tokens into four regimes based on the privilege advantage gap \(A = |\log \Pi_T(y_n|x,p,y_{<n}) - \log \Pi_S(y_n|x,p,y_{<n})|\) and relative predicted probabilities, each token receives a distinct supervision source, strength, and objective. On VLM benchmarks (Qwen3-VL-8B→2B), DOPD outperforms Vanilla OPD by 6.0 points on average across eight benchmarks.

Problem

Standard on-policy distillation (OPD) is bounded by teacher quality. When privileged information (e.g., object bounding boxes for VLMs, verified reasoning hints for LLMs) is added to teacher or student, two distinct gaps are conflated: (1) an intrinsic capability gap, which distillation can close, and (2) an information asymmetry gap, which cannot be replicated without access to privileged inputs. Uniform token-level distillation from a monolithic policy amplifies this "privilege illusion," causing entropy collapse, shortcut fitting, and ultimately worse distillation than vanilla approaches.

Method

DOPD maintains four policy views per training step: a deployed student \(\Pi_S(y|x)\), an on-policy sampled trajectory \(y \sim \Pi_S\), a frozen privileged teacher \(\Pi_T(y|x,p)\), and a parameter-shared privileged student \(\Pi_S(y|x,p)\). The privilege advantage gap \(A_n = |\ell_T - \ell_S|\) (absolute log-prob difference under identical privileged conditions) is computed per token, then normalized within the batch. Combined with the sum of privileged predicted probabilities \(q_T + q_S\), each token falls into one of four regimes:

Regime Condition Objective Rationale
LLH \(A < \bar{A}\), \(q_S+q_T \geq \bar{q}_S+\bar{q}_T\) Top-K reverse KL (teacher, light \(\beta_l\)) Conservative absorption of privileged knowledge
LLL \(A < \bar{A}\), \(q_S+q_T < \bar{q}_S+\bar{q}_T\) Top-K reverse KL (stop-grad student, \(\beta_w < \beta_l\)) Stabilization without imitation of uncertain predictions
IHT \(A \geq \bar{A}\), \(q_T \geq q_S\) Full-vocabulary JS divergence (unit weight) Dense transfer of credible capability advantage
IHS \(A \geq \bar{A}\), \(q_T < q_S\) Top-K reverse KL (stop-grad student, light \(\beta_l\)) Preserve student exploration when teacher signal is unreliable

The total loss is \(\mathcal{L}_{\text{DOPD}} = I_{LH}\mathcal{L}_{LH} + I_{LL}\mathcal{L}_{LL} + I_{HT}\mathcal{L}_{HT} + I_{HS}\mathcal{L}_{HS}\), applied over student on-policy trajectories.

Key Contributions

  • Identification and formalization of "privilege illusion" as a distinct failure mode in OPD with privileged inputs, separating capability gaps from information asymmetry gaps.
  • The privilege advantage gap \(A\) as a per-token proxy for isolating genuine capability discrepancy under controlled privileged conditions.
  • DOPD: a four-regime adaptive routing scheme assigning different supervision sources (teacher vs. self), objectives (KL variants vs. JS), granularity (Top-K vs. full-vocabulary), and intensities per token.
  • Empirical ablations showing high-advantage tokens account for the dominant share of distillation gains (removing them reduces VLM improvement to ~20% of Vanilla OPD).
  • Demonstrated improvements across LLM and VLM settings, continual learning, OOD tasks, and training stability.

Results

  • VLM (Qwen3-VL-8B → Qwen3-VL-2B): DOPD averages 67.6 vs. Vanilla OPD 61.8 (+5.8) and the next-best competitor VA-OPD at ~64.7, across 8 benchmarks including MMMU, MMStar, DynaMath, MathVision, MMMU-Pro, VSI-Bench, RealWorldQA, LCBv5.
  • LLM (Qwen3-8B → Qwen3-1.7B): DOPD averages 67.0 vs. Vanilla OPD 58.0 (+9.0) and next-best Uni-OPD ~60.4, across 8 benchmarks including MATH500, LiveBench, ZebraLogic, AIME25, BFCLv3, C-Eval, AutoLogi, LogicVista.
  • Consistent improvements of 6.2–10.6 points across five model pairs of varying sizes.
  • Token ablation: removing the top-20% high-advantage tokens reduces VLM gains to ~20% of Vanilla OPD at step 100; removing low-advantage or random tokens has negligible effect.

Limitations

  • Requires two additional privileged forward passes (privileged teacher + privileged student) per training step, increasing computational overhead beyond Vanilla OPD.
  • The method assumes the availability of meaningful privileged information; its behavior when privileged inputs are unavailable or of poor quality is less analyzed.
  • The threshold \(\bar{A}\) and coefficients \(\beta_l\), \(\beta_w\) are batch-level averages; the sensitivity analysis is described but exact robustness across varied privileged modalities is not fully characterized in the excerpt.
  • Experiments are confined to the Qwen3 model family; generalization to other architectures is asserted but not shown in the provided text.

Relevance to Vision-Language Models

DOPD directly addresses a failure mode that is particularly acute for VLMs, where privileged inputs such as bounding boxes or region annotations can substantially inflate teacher distributions without providing genuinely learnable signals to the student. The paper validates its approach on Qwen3-VL-8B→2B compression and visual benchmarks (MMMU, DynaMath, VSI-Bench), making it immediately relevant for practitioners training efficient VLMs via distillation. The token-routing framework provides a principled alternative to uniform KL distillation and connects to broader debates in VLM post-training about which visual tokens carry decisive semantic signal. This work complements Vision-OPD and VA-OPD, which introduced structural modifications to handle visual inputs in OPD, by addressing the deeper issue of privileged-information quality under information asymmetry.