Skip to content

Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding

🕒 Published (v1): 2026-05-30 06:34 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; decomposes on-policy distillation gradients for VL visual grounding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Standard on-policy distillation for VLMs conflates language prior matching and visual grounding into a monolithic objective, producing a gradient update that acts as a suboptimal bisector between two nearly orthogonal gradient directions. This paper decomposes the KL loss via Bayes' Rule to isolate the two components, then introduces Visual Gradient Steering (VGS), which explicitly reorients the parameter update toward the visual subspace. VGS outperforms the monolithic baseline across seven multimodal reasoning benchmarks with negligible added compute.

Problem

On-policy distillation for reasoning models has succeeded in text-only settings but transfers poorly to VLMs. Existing approaches (e.g., Qwen3-VL) restrict distillation to text-only data, ignoring visual grounding alignment. When extended naively to the multimodal domain, the standard reverse-KL objective fuses language and visual supervision into one gradient signal that neither fully captures language style nor visual perception—it passively compromises between them rather than resolving the primary bottleneck (visual grounding).

Method

Decomposition. Using Bayes' Rule, the log-likelihood of a multimodal generator factorizes as: $\(\log p(\tau | I, x) = \underbrace{\log p(\tau | x)}_{\text{Language Prior}} + \underbrace{\log p(I|\tau,x)}_{\text{Visual Likelihood}} - \log p(I|x).\)$ This splits the standard on-policy KL objective into two measurable terms: \(L_{\text{Lang}}\) (reverse KL between unimodal student and teacher distributions, sampled on-policy from the full multimodal policy) and \(L_{\text{Vis}}\) (reverse KL against a constructed target \(q_T^*\) that replaces the student's visual likelihood with the teacher's while preserving the student's language prior).

Geometric analysis. For each generated token, the Visual Dependency Score (VDS) \(= D_{\text{KL}}(q_T(\cdot|I,x) \| q_T(\cdot|x))\) is computed. Tokens are binned by VDS; the angle between \(\nabla L_{\text{Lang}}\) and \(\nabla L_{\text{Vis}}\) increases monotonically with VDS, reaching \(\approx 92°\) (near-orthogonality) at the highest bin. The standard gradient \(\nabla L_{\text{Standard}}\) maintains a fixed \(\approx 42°–50°\) offset from both components across all bins—acting as a static bisector.

VGS objective. The steered loss augments the standard loss with a scaled visual term: $\(\ell_{\text{VGS}}(\tau) = \ell_{\text{Standard}}(\tau) + \gamma \ell_{\text{Vis}}(\tau),\)$ then normalizes the gradient magnitude to match that of the original standard gradient (decoupling direction from step size). To prevent "unlearning" of language in the top-30% highest-VDS tokens (where visual and language gradients form obtuse angles), a Language Preservation (LP) regularizer \(\ell_{\text{LP}}\) is added with a small weight \(\lambda = 0.01\), yielding the final objective \(L_{\text{VGS-LP}}\).

Key Contributions

  • First gradient-level decomposition of on-policy VLM distillation, revealing near-orthogonality between language-prior and visual-grounding gradients.
  • Formulation of a tractable visual target distribution \(q_T^*\) that isolates the teacher's visual information gain without requiring explicit visual likelihood computation.
  • Visual Gradient Steering (VGS): a norm-preserving gradient reorientation toward the visual subspace, controlled by a single coefficient \(\gamma\).
  • Language Preservation regularizer applied selectively to high-VDS tokens to prevent catastrophic forgetting of language capabilities under aggressive visual steering.
  • Empirical validation of the Asymmetric Maturity Hypothesis: language prior is already robust; visual grounding is the bottleneck.

Results

All students distilled from Qwen3-VL-8B-Instruct (GRPO-trained on Vision-SR1-47K); evaluated on 7 benchmarks (MathVision, MathVerse-VD/VO, LogicVista, VisualPuzzles, MMMU-Pro-4, VlmsAreBlind).

  • 8B→2B: VGS Acc@1 = 46.10% vs. standard baseline 43.74% (+2.37% average); largest gains on VisualPuzzles (+3.68%) and LogicVista (+3.35%).
  • 8B→4B: VGS Acc@1 = 58.12% vs. baseline 56.64% (+1.56% average); Acc@16 improves by +0.50% on average.
  • Language Steering (symmetric experiment steering toward language instead) consistently degrades performance below baseline, confirming visual grounding as the binding constraint.
  • VGS outperforms the baseline on all seven benchmarks for the 2B student in both greedy (Acc@1) and stochastic (Acc@16) evaluation.

Limitations

  • Experiments are limited to the Qwen3-VL model family (2B/4B students, 8B teacher); generalizability to other VLM architectures is not demonstrated.
  • The steering coefficient \(\gamma\) and LP threshold (70th VDS percentile) require empirical tuning; sensitivity analysis is deferred to appendices.
  • The gradient norm ratio \(\eta_{\text{VGS}}(\gamma)\) is approximated as a fixed constant (0.41 for 2B, 0.36 for 4B) rather than computed dynamically, based on an assumption of stable magnitude ratios during training.
  • The RL + distillation combination is mentioned but not fully presented in the excerpted text.
  • Training data is a single fixed dataset (Vision-SR1-47K); diversity of visual domains is not ablated.

Relevance to Vision-Language Models

This work directly addresses a fundamental training inefficiency in VLM knowledge distillation: the mismatch between language and visual gradient geometries means standard on-policy distillation systematically under-invests in visual grounding, the component that typically lags behind in smaller models. The decomposition into language-prior and visual-grounding components is architecturally general and applies to any autoregressive VLM trained with reverse-KL on-policy distillation. For researchers tracking VLMs, VGS provides a lightweight, theoretically grounded method to improve visual reasoning in compressed models without modifying architecture or data pipelines—complementing RLVR approaches that struggle with cold-start in small models.