Skip to content

OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenVLThinker-7B is an open-source LVLM trained via iterative cycles of lightweight supervised fine-tuning (SFT) and reinforcement learning (RL) to develop R1-style chain-of-thought reasoning in multimodal contexts. The key insight is that SFT acts as an action-highlighting prior that surfaces latent reasoning behaviors (planning, reflection, verification), narrowing the search space for subsequent GRPO-based RL. Three SFT-RL iterations with only 12K training samples yield state-of-the-art performance among open-source 7B reasoning models across six benchmarks.

Problem

Adapting R1-style reasoning to LVLMs is difficult via two existing routes: (1) distilling CoTs from text-only reasoning models (e.g., DeepSeek-R1) into LVLMs via SFT causes performance degradation due to imprecise visual grounding when images are replaced by generated captions; (2) pure RL from scratch faces a prohibitively large search space, making reflective behaviors (self-correction, verification) slow or impossible to emerge in 7B-scale models. Neither approach alone reliably yields complex reasoning in open-source LVLMs.

Method

The method alternates between two steps, iterated three times, always restarting from the base model (Qwen2.5-VL-7B) to prevent overfitting:

  1. Lightweight SFT. In iteration 1, the base LVLM captions each training image; captions plus questions are fed to QwQ-32B (text-only), which generates k reasoning chains via rejection sampling. The shortest correct chain is selected, then truncated at overly repetitive reflection keywords ("Wait", "But wait") to suppress repetitive loops. In later iterations, the previous iteration's LVLM directly generates CoTs on 3K examples, eliminating the caption proxy.

  2. Curriculum GRPO. RL uses Group Relative Policy Optimization with a two-stage source-based curriculum. Data sources (drawn from LLaVA-OneVision / MathV360K overlap) are scored by composite difficulty (z-score average of GPT-4o rating and base model error rate), then k-means clustered into Easy/Medium/Hard. Stage 1 trains on 3K Medium examples; Stage 2 continues on 6K Hard examples (geometry-heavy). This sequence outperforms training on Hard data only.

Each RL stage produces a better LVLM, which then generates higher-quality SFT demonstrations for the next iteration, bootstrapping a self-improvement loop.

Key Contributions

  • Iterative SFT-RL framework that achieves R1-style reasoning in a 7B LVLM using only 12K total training samples (≈1/10 the data of single-iteration concurrent methods).
  • Analysis showing SFT increases frequency of reasoning-action tokens ("wait", "first", "check") while GRPO largely preserves this distribution but increases accuracy (+5.2% on MathVista after GRPO-Iter1).
  • Two-stage source-based curriculum RL using composite difficulty scores (GPT-4o rating + base model error rate).
  • Iterative restarting from base model at each cycle, preventing error accumulation and overfitting.
  • OpenVLThinker-3B variant trained by distilling from the 7B model in a single SFT-RL pass.

Results

  • OpenVLThinker-7B average (6 benchmarks): 47.9% vs. base Qwen2.5-VL-7B 45.6% and best concurrent method VLAA-Thinker-7B 46.6% (trained on 150K samples).
  • MathVista: 72.3% (+3.8% over base 68.5%; VLAA-Thinker: 70.0%; GPT-4o: 63.8%).
  • MathVerse: 50.3% (base: 46.8%; VLAA-Thinker: 48.6%).
  • EMMA: 26.8% (base: not reported comparably; VLAA-Thinker: 25.5%).
  • HallusionBench: 69.1% (+2.7% over base; VLAA-Thinker: 68.4%).
  • MM-Star: 61.9% vs. VLAA-Thinker 55.4%, Qwen2.5-VL 53.9%.
  • WeMath: 64.1% vs. VLAA-Thinker 62.4%, Qwen2.5-VL 61.9%.
  • OpenVLThinker-3B MathVista: 63.4% vs. VLAA-Thinker-3B 61.0%, Qwen2.5-VL-3B 62.3%.
  • Direct RL without SFT prior fails to induce reflection keywords even at equivalent training volume.

Limitations

  • Benchmarks cover mathematical, general, and perceptual reasoning but do not exhaustively probe robustness in other real-world task types.
  • Method validated only at 7B scale; scaling to 32B+ is expected to help but untested due to compute requirements.
  • Initial SFT data relies on image-to-caption conversion via the base LVLM, introducing information loss relative to direct visual input.
  • SFT step at iteration 1 causes a non-trivial accuracy drop before RL recovers it; the iterative approach requires multiple training passes over the base model.

Relevance to Vision-Language Models

This work directly advances the frontier of open-source LVLM reasoning by demonstrating that R1-style chain-of-thought is achievable without proprietary reasoning models or massive datasets, through careful interplay of SFT and RL. The finding that SFT acts as an action-highlighting prior — surfacing latent reasoning tokens rather than directly improving accuracy — is a mechanistically precise insight for the VLM training community. The iterative self-improvement paradigm offers a data-efficient recipe that is directly comparable to concurrent methods (VLAA-Thinker, R1-Onevision) while using 10× less data. The curriculum RL design and analysis of reasoning-keyword dynamics provide concrete engineering guidance for anyone working on multimodal reasoning fine-tuning.