Skip to content

Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SOPHIA is a semi-off-policy RL framework for teaching large vision-language models (LVLMs) slow-thinking multimodal reasoning. It sidesteps the dual failure modes of on-policy RL (bounded by initial policy) and pure off-policy RL (visual hallucinations from cross-model misalignment) by having the LVLM generate its own image descriptions on-policy, then delegating chain-of-thought reasoning to an external LLM. Applied to InternVL3.0-38B, SOPHIA achieves +8.50% average across eight benchmarks, beating GPT-4.1 on MathVision (49.08%) and OlympiadBench (49.95%).

Problem

LVLMs trained primarily on vision-text alignment lack slow-thinking reasoning trajectories in their output distribution, making on-policy RL (e.g., GRPO) fundamentally bounded—it can only amplify existing behaviors. Off-policy RL breaks this ceiling but introduces a new failure: trajectories from external models carry visual features that conflict with the target LVLM's own visual perception, exacerbating hallucinations. No prior approach resolves both limitations simultaneously.

Method

SOPHIA decomposes each trajectory into two parts governed by different policies:

  1. Semi-off-policy sampling: The trainable LVLM (πθ) generates detailed image descriptions on-policy (capturing spatial layouts, semantic relationships, fine-grained details via prompt engineering). An external reasoning LLM (π̄; QwQ-32B for easy, DeepSeek-R1 for hard) then generates slow-thinking reasoning trajectories conditioned on those descriptions—never directly on raw visual tokens it cannot process.

  2. Reward evaluation and propagation: Reasoning trajectories receive outcome-based binary rewards (correct/incorrect). Visual description reward R(cᵢ) is defined as the average correctness over N sampled reasoning trajectories conditioned on that description (Eq. 5). The final dataset reward requires both R(cᵢ) > threshold α and a correct, shortest reasoning trajectory (to combat overthinking).

  3. Policy optimization: Standard off-policy policy gradient with importance sampling. The paper proves (Lemma A.3) that the importance sampling ratio πθ/µ can be approximated with controllable bias using the average reward, simplifying the gradient. The visual transformer (ViT) is frozen during training to preserve vision-language alignment.

  4. Warm-up: Before RL, the LVLM is fine-tuned on image description/captioning data (RLAIF-V, WildVision) to improve the quality of on-policy descriptions feeding the behavior model.

Key Contributions

  • Identifies and resolves the conflict between on-policy distribution limits and off-policy visual misalignment in LVLM reasoning RL.
  • Semi-off-policy behavior model: on-policy visual understanding + off-policy LLM reasoning, requiring no closed-source or human-annotated data.
  • Reward backpropagation from reasoning outcomes to visual descriptions, incentivizing accurate visual grounding.
  • Theoretical justification showing the IS ratio reduces to an average-reward approximation with bounded bias.
  • Empirical demonstration that SOPHIA provides a better policy initialization for subsequent on-policy RL (SOPHIA+GRPO outperforms GRPO alone and avoids performance collapse).

Results

  • InternVL3.0-38B + SOPHIA: average +8.50% over base across 8 benchmarks; 55.51% average pass@1.
  • MathVision: 49.08% (+16.15% over base InternVL3.0-38B at 32.93%; surpasses GPT-4.1 at 38.29%).
  • OlympiadBench: 49.95% (+24.45% over base at 25.50%; surpasses GPT-4.1 at 30.24% and Qwen2.5-VL-72B at 35.86%).
  • MMMU Pro: 52.60% vs. MPO 49.08% and GRPO 49.88%.
  • InternVL2.5-38B + SOPHIA: average +10.94% over base (43.09% → 54.03%).
  • SOPHIA+GRPO (continued on-policy): 57.28% average, reaching Claude-3.7-Sonnet level on college-level tasks, surpassing most closed-source models on scientific reasoning.
  • Hallucination reduction (qualitative, 50 MMMU Pro samples): visual hallucination 38% → 16%; logic inconsistency 54% → 44%.
  • 8B scale: InternVL3.0-8B+SOPHIA reaches 44.71% average, best among 8B models on MMMU and OlympiadBench; worse on MathVista (attributed to underrepresented problem types in training data).
  • Geometry (DynaMath/MathVision/OlympiadBench): SOPHIA outperforms all open-source models, e.g., OlympiadBench solid geometry: 62.81% vs. GPT-4.1's 13.65%.
  • Ablations confirm caption reward, shortest-trajectory selection, and warm-up each contribute positively; freezing ViT consistently improves average performance.

Limitations

  • Struggles with long-range visual dependencies and fine-grained/low-level recognition in complex scenes.
  • Performance degrades for underrepresented problem types or domains (e.g., age-difference reasoning at 8B scale, MathVista performance dips at 8B).
  • Binary reward signal is coarse; no dense step-level supervision.
  • Visual hallucinations and redundant reasoning still persist after training (not fully eliminated).
  • Training data quality and difficulty matter more than quantity; private 80K dataset outperforms larger public MathV360K subsets, limiting full reproducibility.
  • Scaling law shows diminishing returns on math-perception tasks (MathVista, MathVerse, DynaMath) with more data; not always monotone.

Relevance to Vision-Language Models

SOPHIA directly addresses a fundamental training challenge for LVLMs: how to acquire slow-thinking reasoning when vision-text pretraining leaves models without such capabilities in their initial distribution. The semi-off-policy design is architecturally significant—it establishes a principled way to leverage stronger text-only reasoning models (DeepSeek-R1, QwQ) to bootstrap LVLM reasoning without importing their visual hallucinations, a problem that prior off-policy distillation approaches have struggled with. The reward backpropagation mechanism and ViT-freezing findings are directly actionable for practitioners fine-tuning LVLMs with RL. For the broader VLM community, SOPHIA's framing—that visual understanding and reasoning should be trained with matched policy distributions—offers a principle applicable beyond math tasks to any domain requiring grounded multimodal chain-of-thought.