Skip to content

Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Chain-of-Step (CoS) decomposes VLM reasoning chains into structured, named steps with explicit reflection components, then trains a Process Reward Model (PRM) to score individual steps rather than only final answers. This fine-grained reward signal drives iterative DPO and step-level beam search, yielding consistent accuracy gains across six multimodal benchmarks over strong baselines.

Problem

Existing VLM chain-of-thought approaches produce unstructured, coarse-grained reasoning blobs that are hard to evaluate at intermediate steps, making it difficult to assign accurate rewards for RL training or to perform effective inference-time scaling. Without step-level reward signals, RL optimizes only on final-answer correctness, missing quality signals within the reasoning chain.

Method

Structured reasoning template. Each reasoning chain is decomposed into consecutive steps, each containing three sub-components delimited by special tokens: Name (step summary), Thought (detailed inference), and Reflection (connection to visual content and prior steps). Special tokens <|reasoning_start|>, <|reasoning_proceed|>, and <|reasoning_end|> enforce format stability without relying on fragile prompt engineering.

ShareGPT-Step-300K. GPT-4o is prompted with question + ground-truth answer to generate step-structured reasoning for 300K instances across 17 tasks (math, science, chart/document, world knowledge). SFT on this dataset teaches the base VLM to produce CoS-formatted outputs.

Process Reward Model (PRM). Step-level correctness annotations are generated via (a) Monte Carlo estimation (Math-Shepherd, N=16 rollouts) and (b) GPT-4o-as-Judge with Good/Neutral/Bad labels, totaling 200K annotated steps. InternVL-2.5-MPO-38B is fine-tuned as the PRM using binary cross-entropy on these labels.

Iterative DPO with fine-grained rewards. The SFT model generates 16 candidate reasoning paths per question; the PRM scores each at the step level. The final score is a weighted sum of average step score (weight 20% by default, selected via ablation) and final-answer score, normalized to [0,1]. Positive/negative pairs are selected if they exceed a score gap threshold, and three rounds of DPO are run with a fixed reference model (the SFT model).

Step-level beam search. At inference, N candidates for step 1 are scored by PRM; the best step is extended, and this greedy process repeats until the answer is generated—matching the token cost of Best-of-N while achieving strictly better accuracy.

Key Contributions

  • A structured three-component step format (Name/Thought/Reflection) with special-token enforcement for stable VLM output.
  • ShareGPT-Step-300K: 300K GPT-4o-generated step-level reasoning instances spanning 17 tasks (to be released publicly).
  • A 38B PRM trained on 200K dual-annotated (MC + LLM-Judge) step labels, achieving 87.3% step accuracy on unseen questions.
  • Iterative DPO training regime using combined step+answer PRM scores, outperforming outcome-only RL.
  • Step-level beam search enabling inference-time scaling at the same sampling cost as Best-of-N.
  • Empirical finding: for visual reasoning, reasoning quality matters more than length; PRM-guided RL reduces average step length before gradually increasing it, unlike outcome-only RL which monotonically increases length.

Results

  • CoS on InternVL-2.5-MPO-8B: average score 73.4 vs. baseline 69.4 (+4.0 pp) across MathVista, MMStar, MMMU, M3CoT, AI2D, ChartQA.
  • CoS on LLaVA-NeXT-8B: average 64.2 vs. baseline 52.1 (+12.1 pp); SFT alone +8.7 pp, iterative DPO adds further +3.4 pp.
  • Reward ablation (LLaVA-NeXt, M3CoT): Step+Answer PRM → 71.7%; Outcome → 70.0%; Answer-only PRM → 69.7%.
  • Reasoning pattern ablation (average across MathVista/MMStar/M3CoT): CoS RL 61.8 vs. Direct Prompt RL 60.2 (+1.6) vs. No-Reason RL 57.1; CoS SFT→RL gain (+4.0) exceeds Direct Prompt gain (+2.7).
  • GRPO ablation: CoS GRPO 63.0 average vs. outcome GRPO 61.2 vs. SFT baseline 57.8.
  • Inference-time scaling (M3CoT, N=64): PRM-BS > PRM Best-of-N > Self-Consistency > greedy; PRM Best-of-N substantially exceeds Self-Consistency at all N.
  • PRM accuracy on unseen data: InternVL-2.5-38B achieves 87.3% step accuracy and 88.7% answer accuracy.

Limitations

  • The PRM is a 38B model, imposing substantial inference cost for both training-time reward scoring and inference-time beam search.
  • ShareGPT-Step-300K is synthetically generated by GPT-4o from ground-truth answers ("reasoning from results"), which may introduce annotation bias and limit coverage of novel problem types.
  • Evaluation is limited to six benchmarks concentrated on math, science, and chart domains; generalization to open-ended or perception-heavy tasks is not assessed.
  • Step-level beam search requires structured CoS output, making the approach incompatible with standard CoT models without retraining.
  • The optimal step-score weight (20%) and DPO margin threshold are tuned on specific benchmarks and may not transfer.

Relevance to Vision-Language Models

CoS directly addresses a known weakness in VLM reasoning research: the lack of intermediate process supervision analogous to what Math-Shepherd provides for text-only LLMs. By formalizing step decomposition with a Reflection sub-component that explicitly grounds each step in visual content, the work introduces a VLM-specific design choice that mitigates hallucination during multi-step reasoning. The finding that reasoning length and quality decouple in the visual domain—unlike in pure math LLMs where longer chains correlate with better performance—is an important empirical signal for the field. The released dataset (ShareGPT-Step-300K), PRM, and code provide concrete infrastructure for future work on process reward models and inference-time scaling in multimodal settings.