VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VL-Rethinker applies reinforcement learning to instill slow-thinking (self-reflection) capabilities in vision-language models without distillation from stronger teacher models. It introduces two techniques—Selective Sample Replay (SSR) to fix GRPO's vanishing advantages problem, and Forced Rethinking to explicitly train self-correction behavior. The resulting 72B model sets open-source SOTA on MathVista (80.4%), MathVerse (63.5%), and several multi-discipline benchmarks, surpassing OpenAI-o1 on math-focused tasks.

Problem¶

Slow-thinking systems (GPT-o1, DeepSeek-R1) vastly outperform fast-thinking models on text-only reasoning but show no such advantage in multimodal settings—GPT-o1 scores only 73.9%/57.0% on MathVista/MathVerse, on par with fast VLMs like Qwen2.5-VL-72B. Standard GRPO-based RL for VLMs suffers from vanishing advantages (uniform reward within a group → zero gradient signal), and RL alone does not reliably induce explicit self-reflection in VLMs unlike in text LLMs.

Method¶

Two-stage RL pipeline on Qwen2.5-VL-Instruct:

Stage 1 — GRPO with Selective Sample Replay (SSR): SSR augments each GRPO training batch by replaying high-value past experiences from a replay buffer. The buffer stores only samples from query groups that exhibited non-zero advantages (|Â| > 0). Samples are drawn with probability proportional to |Â|^α, prioritizing examples near the model's decision boundary. This redistributes the advantage distribution away from the zero-mass peak that plagues standard GRPO, stabilizing training on 38,870 curated multimodal queries.

Stage 2 — Forced Rethinking: After the initial rollout y₁, a textual "rethinking trigger" (one of three types: self-verification, self-correction, self-questioning) is appended, and the model generates y₂. The full sequence y = y₁ ⊕ trigger ⊕ y₂ is used. Only successful rethinking trajectories (those yielding a correct final answer) are retained, and an additional SFT imitation loss trains the model to internalize this behavior. At inference, the trained model adaptively decides whether to rethink rather than always triggering.

Key Contributions¶

Identification and characterization of the vanishing advantages problem in GRPO for VLMs, with empirical evidence that effective-query ratio drops from ~40% to <20% within 256 steps on 72B models.
Selective Sample Replay (SSR): priority-weighted experience replay integrated into GRPO to maintain non-zero gradient signal throughout training.
Forced Rethinking: training-time rollout augmentation with typed trigger tokens + SFT loss on successful rethinking traces to explicitly cultivate metacognitive self-correction.
Empirical finding that VLM "slow thinking" is qualitatively distinct from LLM slow thinking—accuracy gains do not correlate with longer reasoning chains but with perceptual re-verification ("look twice, not think longer").
Release of code, models, and a 39K high-quality multimodal training dataset.

Results¶

VL-Rethinker-72B on MathVista: 80.4% (+5.6% over open-source SOTA Qwen2.5-VL-72B at 74.8%; beats GPT-o1 at 73.9%)
VL-Rethinker-72B on MathVerse: 63.5% (+6.3% over prior open SOTA; beats GPT-o1 at 57.0%)
VL-Rethinker-72B on MathVision: 44.9% (+6.5% over open SOTA)
VL-Rethinker-72B on MMMU-Pro: 55.9% (+3.7% over open SOTA)
VL-Rethinker-72B on EMMA: 38.5% (+4.4% over open SOTA)
VL-Rethinker-72B on MEGA-Bench: 51.3% (+2.3% over open SOTA)
VL-Rethinker-7B: outperforms all 7B RL-trained competitors (e.g., OpenVLThinker-7B 70.2% → 74.9% on MathVista; R1-OneVision-7B 46.4% → 54.2% on MathVerse)
Ablation: GRPO-SSR > GRPO-Filter (=DAPO) > standard GRPO on all benchmarks; SSR prevents the overfitting / reward degradation seen in baseline GRPO

Limitations¶

Still lags behind human expert performance on open-ended real-world tasks (EMMA, MEGA-Bench); authors attribute this to insufficient high-quality training data.
The rethinking mechanism is motivated primarily by mathematical/visual reasoning; generalization to broader multimodal tasks (e.g., MMMU val: 68.8%, slightly below Qwen2.5-VL-72B at 69.4%) remains incomplete.
Training is resource-intensive: VL-Rethinker-72B required ~60 hours on 64× A800 (80G) GPUs.
Forced Rethinking applies SFT only on successful rethinking traces, introducing a survivorship bias that may underweight failure correction signals.
The replay buffer design (hyperparameter α, buffer size K) is not extensively ablated in the main paper.

Relevance to Vision-Language Models¶

VL-Rethinker directly advances the frontier of VLM reasoning by demonstrating that slow-thinking behavior—previously elusive in the multimodal domain despite being potent in text-only LLMs—can be reliably induced via RL without distillation. The key insight that multimodal "rethinking" is fundamentally about perceptual re-verification rather than extended logical deliberation reframes how the community should think about improving VLM reasoning: the bottleneck is visual grounding accuracy, not chain-of-thought depth. SSR is a broadly applicable fix to the vanishing-advantages failure mode that affects any GRPO-based VLM training, and Forced Rethinking provides a lightweight, distillation-free alternative to SFT pipelines that depend on teacher models. Together, these contributions are directly relevant to ongoing work on RL alignment of VLMs, test-time compute scaling in multimodal settings, and self-correction mechanisms.