Skip to content

SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning

๐Ÿ•’ Published (v1): 2025-01-01 ยท Source: NeurIPS ยท Venue: NeurIPS 2025 ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

SRPO is a two-stage reinforcement learning framework that teaches multimodal LLMs to explicitly self-reflect and self-correct during both supervised fine-tuning and RL training. It constructs a 10K reflection-focused SFT dataset via GPT-o4-mini distillation, then applies a GRPO variant with a reflection-aware reward that incentivizes concise, functionally useful reflections. On Qwen-2.5-VL-7B/32B, SRPO consistently outperforms all open-source reasoning MLLMs across math and general multimodal benchmarks.

Problem

Existing RL-based multimodal reasoning methods (e.g., GRPO, VL-Rethinker, MM-Eureka) treat reasoning as a token-level Markov process with local dependencies, producing redundant, repetitive, or erroneous intermediate steps. Prior self-reflection attempts via prompting or vanilla RL are bounded by the model's pre-training knowledge distribution and cannot inject genuinely new reflective behaviors. No existing method simultaneously incentivizes structured self-reflection in both the SFT cold-start and RL phases.

Method

Stage 1 โ€” Reflection-oriented SFT cold-start: - Sample 10K multimodal reasoning problems from LLaVA-CoT, Mulberry, and MathV360K. - For each problem, generate the policy model's initial CoT response; then use GPT-o4-mini with the ground truth to produce a <reflection> that either (a) corrects an incorrect CoT (~70% of samples) or (b) streamlines a verbose correct CoT (~30%). - Train the policy model with a cross-entropy loss over the sequence: initial response โ†’ <reflection> โ†’ ground-truth answer.

Stage 2 โ€” Reflection-aware GRPO (SRPO): - Output format: <think> (first solution) โ†’ <reflection> โ†’ <think> (refined solution) โ†’ <answer>. - Total reward: R_total = R_task + R_reflection. - R_task: format reward (0.5 for correct <think> tags) + accuracy reward (0.5 if first solution matches ground truth). - R_reflection: format indicator I_ref (0.25 for proper <reflect> tags) + effectiveness indicator I_eff (+0.5 if reflection fixes wrong answer, +0.25 if it preserves correct, 0 if it fails, โˆ’0.25 if it corrupts correct) + brevity reward f_len = exp(โˆ’((L โˆ’ T_target)/(T_max โˆ’ T_target))ยฒ) peaking at T_target = 2ร— original response length and decaying to zero at T_max = 2.5ร—. - RL training on 37K cross-domain samples (ScienceQA, ChartQA, AI2D, MATH, MMK12, PhyX, etc.) for 3 epochs with OpenRLHF, 8 rollouts per sample, lr=1e-6.

Key Contributions

  • Reflection-oriented SFT data pipeline: uses a strong MLLM teacher to generate structured <reflection> segments that teach both error correction and CoT conciseness.
  • Reflection-aware GRPO reward: I_eff directly measures whether reflection functionally improves the second answer; f_len softly penalizes verbosity with a smooth exponential; together they prevent reflection gaming (empty or inflated reflections).
  • Empirical demonstration that reflection-enhanced RL generalizes to unseen cross-disciplinary tasks (physics, chemistry, biology in MMK12) not present in training.
  • Analysis showing self-reflection integration improves PPO and DAPO as well, not just GRPO.

Results

  • SRPO-7B vs. GRPO-7B (Qwen-2.5-VL-7B base): MathVista 75.8 vs. 72.3 (+3.5), MathVerse 55.8 vs. 52.9 (+2.9), MMMU-Pro 42.3 vs. 39.9 (+2.4), Physics (MMK12) 60.6 vs. 53.5 (+7.1).
  • SRPO-32B achieves MathVista 78.5, MathVerse 58.9, MathVision 39.6, MMMU-Pro 51.3 โ€” best among all open-source models in Table 1.
  • SRPO-32B exceeds Gemini2-flash by 4.6 on EMMA.
  • SRPO-7B on MMK12 cross-disciplinary: 75.3 Math / 60.6 Physics / 70.3 Chem / 69.5 Bio โ€” surpasses MM-Eureka-7B by 5.1 on Physics and OpenVLThinker-7B by 9.7 on Chemistry.
  • Ablation: removing Self-Reflection SFT drops avg from 53.5 to 51.2; removing Self-Reflection RL collapses avg to 46.6 (near base model), demonstrating RL stage is critical.
  • 2-step thinking (GRPO + two <think> blocks) yields only 50.3 avg vs. SRPO's 53.5, confirming that structured reflection is not equivalent to additional thinking steps.

Limitations

  • SFT dataset is only 10K samples, and reflection quality depends on the availability and cost of a strong teacher model (GPT-o4-mini).
  • Reflection-aware reward hyperparameters (T_target, T_max, ฮฑ) require tuning; sensitivity analysis is limited to RL data-size ablations.
  • Evaluated only on Qwen-2.5-VL-7B and 32B; generalization to other MLLM families (e.g., LLaMA-Vision, InternVL) is not demonstrated.
  • The approach requires two distinct training stages, increasing total compute vs. single-stage RL methods.
  • No analysis of failure modes where reflection worsens performance at inference time on out-of-distribution inputs.

Relevance to Vision-Language Models

SRPO directly targets a core weakness in current VLMs: the inability to detect and repair their own reasoning errors during multi-step visual-language tasks. By framing self-correction as a learnable, reward-shaped behavior rather than a post-hoc prompting trick, it advances a principled path beyond chain-of-thought scaling for open-source MLLMs. The reflection-aware reward design โ€” particularly I_eff, which penalizes reflections that corrupt correct answers โ€” is a reusable component that could augment other VLM RL pipelines (PPO, DAPO). This work is closely related to the broader effort to close the gap between open-source multimodal reasoning models and proprietary systems like GPT-o1/o3 on math-heavy and cross-disciplinary VQA benchmarks.