Skip to content

Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FAST-GRPO modifies the GRPO reinforcement learning algorithm for large vision-language models (LVLMs) to dynamically adapt reasoning chain length based on estimated question difficulty, replacing the uniform "slow thinking" paradigm. It combines difficulty-aware length rewards and adaptive KL regularization to achieve >10% accuracy gains over the base model while cutting token usage 32.7–67.3% versus prior slow-thinking methods.

Problem

Existing RL-based LVLM reasoning methods fall into two failure modes: RL-only methods (e.g., MM-R1, MM-Eureka) fail to meaningfully scale reasoning length (only −20% to +10% vs. base), while SFT-RL two-stage methods (e.g., R1-OneVision) exhibit pathological overthinking—producing chains ~2× longer than the base model uniformly across all difficulty levels, which actively harms accuracy on easy questions (69.5% vs. 72.7% baseline on Geometry 3K easy split). No prior work effectively balanced fast and slow thinking for multimodal inputs.

Method

FAST-GRPO extends GRPO with three coupled components:

  1. Multimodal difficulty estimation. Two metrics are combined multiplicatively: extrinsic difficulty (S_ext = 1 − pass@k, computed online from k rollouts) captures model-relative hardness; intrinsic image complexity (H_image) combines GLCM texture entropy over image patches and ViT classification entropy from the final feature layer to capture visual semantic complexity. Final score: S_difficulty = S_ext × H_image.

  2. Difficulty-aware length reward (r_t). For questions below the 80th-percentile difficulty threshold θ, correct responses are rewarded proportionally to how much shorter than average they are (1 − L/L_avg). For hard questions, incorrect responses are rewarded for being longer (min(L/L_avg − 1, 1)). Correct hard responses and all other cases receive zero length reward, preventing unconstrained verbosity.

  3. Difficulty-aware KL regularization. The KL coefficient is set per-sample as β_d = β_min + (β_max − β_min)(1 − S_ext), giving hard questions lower KL penalty (more exploration) and easy questions higher regularization (stability and OOD generalization).

Slow-to-Fast curriculum sampling filters training data by difficulty across epochs: early epochs exclude easy samples (strengthen hard reasoning), later epochs exclude hard samples (practice concise reasoning). Two variants tested: binary (two discrete phases) and continuous (smooth probability shift); binary performs slightly better.

Key Contributions

  • First empirical characterization of fast-slow thinking feasibility in LVLMs, showing length rewards decouple from accuracy (±3% accuracy across 180–700 token range) and data difficulty acts as implicit length regulator.
  • Multiplicative multimodal difficulty metric combining online extrinsic pass@k rates with GLCM + ViT entropy for visual complexity.
  • Difficulty-conditioned length reward that rewards brevity on easy correct responses and thoroughness on hard incorrect ones.
  • Per-sample adaptive KL coefficient β_d that modulates exploration vs. exploitation by question difficulty.
  • Slow-to-Fast binary curriculum sampling that trains on hard questions first, then easy ones.

Results

  • FAST-7B vs. base (Qwen2.5-VL-7B): >10% average accuracy improvement across 7 benchmarks; MathVista 73.8 vs. 68.2, MathVerse 50.6 vs. 46.9, MathVision 30.6 vs. 25.6.
  • FAST-7B vs. R1-OneVision (SFT-RL): 67.3% shorter responses on average while matching or exceeding accuracy (e.g., MathVision 30.6 vs. 29.9, MathVerse 50.6 vs. 46.4).
  • FAST-7B vs. MM-R1 (RL-only): 32.7% shorter responses; MathVerse 50.6 vs. 49.8.
  • vs. closed-source: MathVista 73.8 and MathVerse 50.6 surpass GPT-4o (63.8 and 41.9).
  • FAST-3B: >14% average accuracy improvement over Qwen2.5-VL-3B base.
  • vs. fast-thinking reward baselines (Kimi, CosFn, DAST): FAST achieves highest average accuracy (59.0 vs. 57.8/56.7/55.7) with competitive length (162.2 avg tokens vs. DAST's 156.9).
  • Cross-domain: 7–9% improvement on MM-K12 science reasoning; 15–25% shorter responses on open-domain VQA (Bingo, MMHal).
  • Adaptive behavior confirmed: Hard Geometry 3K questions produce 60% longer responses than easy ones; MM-Math responses average 79% more tokens than the easier benchmarks.
  • Ablation: Removing data sampling most hurts accuracy; removing thinking reward increases length by ~42% relatively; removing difficulty-aware regularization costs 1.8 pts on MathVista.

Limitations

  • Image complexity metric H_image shows only moderate correlation with human difficulty judgments (SRCC=0.49, PLCC=0.54 vs. GLCM alone at 0.75/0.77), meaning difficulty estimation is imprecise.
  • The multiplicative difficulty score assigns low difficulty to questions that are visually simple but hard for the model; these edge cases (< 5% of training data) receive zero length reward and are not actively handled.
  • Dominant failure mode (~50%+ of errors) is visual perception (misread scales, chart values, spatial relations), which the adaptive reasoning approach cannot correct—it requires better OCR and grounding.
  • Experiments confined to Qwen2.5-VL backbone; generalization to other LVLM architectures not demonstrated in the main paper (32B results in appendix only).
  • Training data limited to verifiable closed-form (multiple-choice, numeric) questions; open-ended generation tasks are not evaluated.

Relevance to Vision-Language Models

FAST-GRPO directly addresses a key bottleneck in applying chain-of-thought RL to VLMs: the overthinking problem induced by SFT on distilled slow-thinking data, which is a widely used paradigm (R1-OneVision, OpenVLThinker, Vision-R1). By demonstrating that reasoning length and accuracy are largely decoupled under length rewards in VLMs—a finding that differs from text-only LLMs—the paper motivates difficulty-adaptive training as a principled alternative to uniform slow thinking. The incorporation of visual complexity (GLCM + ViT entropy) as part of difficulty estimation is a novel, VLM-specific design that goes beyond length penalty methods developed for text-only models, making this work directly relevant to anyone training or fine-tuning VLMs with RL.