SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Self-distilled cold start decouples visual/language learning for RLVR VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPECS introduces a three-stage cold-start framework for reinforcement learning on vision-language models that replaces standard SFT warm-up with preference-based (DPO) pre-alignment. The key insight is that SFT cold start causes instruction-style overfitting and poor OOD generalization, whereas DPO on self-distilled format-focused preference data produces a better initialization for subsequent RLVR. On Qwen2.5-VL-7B, SPECS improves MEGA-Bench by 4.1% and MathVista by 12.2% over strong baselines.

Problem

SFT-based cold start—the dominant warm-up strategy before RLVR training—jointly learns reasoning content, format, and style, leading to instruction-style overfitting, degraded OOD generalization, and a suboptimal initialization that limits the performance ceiling of downstream RL. Additionally, teacher-distillation approaches for generating cold-start data suffer when the teacher–student capability gap is large.

Method

SPECS is a three-stage pipeline applied to a base VLM (Qwen2.5-VL-7B):

1. Self-distillation for preference data generation. A brief GRPO run on the base model yields \(\pi_{GRPO\text{-}zero}\), which then generates responses alongside the base model. Chosen responses (\(y^+\)) are filtered by Gemini-2.5 Flash for reasoning–answer consistency. Rejected responses (\(y^-\)) share the correct final answer but have deliberately corrupted formatting (five corruption strategies: tag removal variants). This ensures the preference signal is purely about output format, not answer correctness.

2. DPO-based pre-alignment (cold start). The base model is fine-tuned with a hybrid loss: $\(\mathcal{L}_{hybrid} = \mathcal{L}_{DPO} + \lambda \mathcal{L}_{SFT}\)$ where \(\mathcal{L}_{DPO}\) is the standard DPO loss over the self-distilled pairs and \(\mathcal{L}_{SFT}\) is negative log-likelihood on chosen responses (\(\lambda=1\)). This yields a format-aligned "Warmup Model."

3. Final GRPO fine-tuning. Starting from the Warmup Model, GRPO is applied with composite reward \(R_{total} = R_{format} + R_{acc}\), where \(R_{format}=0.5\) for structurally correct output and \(R_{acc} \in \{0,1\}\) determined rule-based for MC/numerical or GPT-4o-judged for short-answer.

To quantify generalization, the paper introduces the Generalization Factor (GF) \(\Gamma(n)\), an \(F_\beta\)-score (\(\beta=2\)) combining ID and OOD performance gains over a baseline: $\(\Gamma(n) = (1+\beta^2)\frac{G_{ID}(n)\cdot G_{OOD}(n)}{\beta^2 \cdot G_{ID}(n) + G_{OOD}(n)}\)$

Key Contributions

• The SPECS three-stage cold-start framework that decouples format learning (DPO) from reasoning learning (RLVR).
• The Generalization Factor (GF) metric for quantifying cold-start method quality via ID/OOD performance balance.
• Empirical demonstration that DPO (especially DPO+SFT loss) generalizes better than SFT as cold start, as measured by GF across training steps.
• A self-distillation pipeline that avoids reliance on larger teacher models: \(\pi_{GRPO\text{-}zero}\) provides chosen responses, and format corruption generates rejected responses with correct answers.
• Ablation evidence that decoupled preference data (format-only signal) outperforms coupled data (mixed format + correctness signal) for cold-start DPO.

Results

• MEGA-Bench Core: Ours-7B = 42.64 vs. backbone (Qwen2.5-VL-7B) = 38.84 (+3.8 pp); vs. best reasoning baseline Orsta-7B = 41.65 (+4.1 pp over backbone-comparable).
• MathVista: Ours-7B = 75.90 vs. backbone = 63.70 (+12.2 pp); vs. best baseline VL-Rethinker-7B = 73.60.
• MathVerse (vision only): Ours-7B = 48.73 vs. backbone = 38.20 (+10.5 pp).
• MMMU val: Ours-7B = 56.78 vs. backbone = 54.2 (+2.5 pp).
• MathVision: Ours-7B = 29.50 vs. backbone = 25.40 (+4.1 pp).
• Ablation: Self-distillation (avg 50.02) outperforms Qwen72B-teacher distillation (avg 48.98) and Qwen32B-teacher distillation (avg 46.43) across five benchmarks.
• Ablation: Decoupled data (avg 50.02) outperforms coupled data (avg 48.68).

Limitations

• The three-stage pipeline adds training complexity and compute overhead compared to a single SFT cold start; Stage 1 requires a full GRPO run just to generate preference data.
• Relies on Gemini-2.5 Flash as an external evaluator for chosen-response filtration, introducing a dependency on a proprietary model.
• Evaluated only on a single base model family (Qwen2.5-VL-7B); generalizability to other VLM architectures is unverified.
• The GF metric uses fixed \(\beta=2\) weighting, and sensitivity to this hyperparameter in other settings is not fully analyzed in the main text.
• Format corruption strategies for rejected responses are heuristic; edge cases where the corruption inadvertently changes answer correctness are not fully discussed.

Relevance to Vision-Language Models

SPECS directly addresses a core bottleneck in the emerging "MLLM-r1" paradigm—how to properly initialize VLMs before RLVR—by showing that format alignment and reasoning ability should be learned in separate stages. This is consequential for VLM training pipelines because SFT cold start is the de facto standard and SPECS provides a drop-in replacement with measurable gains on both general multimodal benchmarks (MEGA-Bench) and mathematical reasoning benchmarks (MathVista, MathVerse). The self-distillation mechanism is particularly relevant as it removes dependence on proprietary teacher models, lowering the barrier to reproducing and extending MLLM-r1 style training. The introduced GF metric also offers a principled diagnostic tool for comparing cold-start strategies that could be broadly adopted in VLM training research.