Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Reason-RFT is a two-stage post-training framework for VLMs that first activates reasoning via CoT supervised fine-tuning, then applies GRPO-based reinforcement learning to improve generalization under domain shift. It addresses the overfitting and cognitive rigidity induced by pure CoT-SFT by replacing static imitation with reward-driven exploration. On visual counting, structure perception, and spatial transformation benchmarks, it outperforms both SFT-only and RL-only baselines, particularly under distribution shift.

Problem

Chain-of-Thought SFT improves VLM reasoning but causes overfitting and cognitive rigidity, limiting transfer to out-of-distribution (domain-shift) visual inputs. Pure RL-from-scratch methods lack stable initialization for complex structured outputs and exhibit a "transient adaptation gap" early in training. Neither paradigm alone achieves robust generalization with limited labeled data.

Method

Two sequential stages applied to Qwen2-VL (2B and 7B):

1. Stage 1 — SFT-based Activation: Fine-tune on curated CoT data (step-by-step reasoning traces) to initialize the model with domain-specific reasoning priors. The objective maximizes log-likelihood over concatenated reasoning and answer tokens.

2. Stage 2 — GRPO-based RL Enhancement: Sample G candidate responses per input from the Stage 1 policy; compute task-specific rewards; normalize rewards within the group to relative advantages; update via policy gradient with KL-divergence regularization against the reference policy.

Reward design has two components: - Format reward: binary reward for adherence to <think>...</think><answer>...</answer> structure. - Accuracy reward: task-specific — discrete exact-match for counting/multiple-choice; cosine-interpolated tolerance reward for numerical/LaTeX answers (with thresholds ε₁=0.05, ε₂=0.20); partial+full sequence-match reward for spatial transformation function sequences (weighted by coefficients α, β).

The benchmark dataset is reconstructed from CLEVR-Math (35K visual counting), Geo170K/Math360K (4.5K structure perception), and TRANCE (60K spatial transformation), with domain-shift test splits from Super-CLEVR and novel viewpoints.

Key Contributions

• First two-stage RFT framework combining CoT-SFT initialization with GRPO-based RL for visual reasoning in VLMs.
• Systematic ablation comparing ANS-SFT, CoT-SFT, Reason-RFT-Zero (RL only), and Reason-RFT (SFT+RL) across in-domain and domain-shift settings.
• Three task-specific reward mechanisms (discrete, mathematical, function-based) enabling verifiable RL training across heterogeneous visual reasoning tasks.
• Curated benchmark spanning visual counting, structure perception, and spatial transformation with explicit domain-shift evaluation splits.
• Identification and analysis of three training dynamics: Greedy Reward Stratification, Transient Adaptation Gap, and Reasoning Redundancy.

Results

• Visual counting (DS avg, 7B): Reason-RFT outperforms ANS-SFT by 13.93% under domain shift.
• Structure perception (DS, 2B): Reason-RFT achieves 6.93% gain over CoT-SFT.
• Spatial transformation (DS avg, 2B): Reason-RFT exceeds CoT-SFT by 21.04% and surpasses GPT-4o by 31.7%.
• Data efficiency: Reason-RFT (2B) reaches 70% of Reason-RFT-Zero's final performance with 3% of training data (≈1,600 samples); 82.5% with 9% of data. The 7B variant exceeds 92% of Reason-RFT-Zero using 3% of data.
• ID performance: Competitive with or superior to InternVL-2.5-8B, GPT-4o, and Gemini-1.5-Pro on most tasks in the 7B setting.
• Hard vs. soft rewards: Penalized partial-match rewards (α=−0.25, β=−0.50) improve DS generalization over partial-credit rewards in spatial transformation.

Limitations

• Reasoning Redundancy: Stage 1 CoT data distilled from GPT-4o induces verbose reasoning traces that persist through Stage 2; Reason-RFT generates significantly longer chains than Reason-RFT-Zero at similar accuracy, indicating potential overthinking and unnecessary compute overhead.
• Domain coverage is narrow — three task types (counting, geometry perception, spatial transformation); generalization to open-domain VQA or OCR-heavy tasks is unverified.
• No explicit length penalty or conciseness reward in GRPO; controlling trace verbosity is left as future work.
• Smaller models (2B) rely more heavily on Stage 1 CoT priors, suggesting diminished RL benefit when CoT data quality or coverage is limited.
• Transient Adaptation Gap in Reason-RFT-Zero shows pure RL is brittle early in training, requiring careful initialization or curriculum.

Relevance to Vision-Language Models

Reason-RFT directly advances the post-training paradigm for VLMs by showing that GRPO-style RL — already successful in text-only reasoning (DeepSeek-R1, GPT-o1) — can be adapted for multimodal tasks with carefully engineered, task-specific verifiable rewards. The finding that CoT-SFT is a necessary but insufficient step (SFT+RL > SFT-only and RL-only) provides a concrete recipe for practitioners fine-tuning VLMs on structured visual reasoning. The Reasoning Redundancy finding is a cautionary result relevant to anyone studying chain-of-thought behavior in VLMs: SFT on distilled CoT data encourages verbosity that RL alone does not suppress, motivating future work on length-controlled RLVR. The benchmark and domain-shift evaluation protocol also serve the broader VLM evaluation community.