Skip to content

Vision-SR1: Self-Rewarding Vision-Language Model via Reasoning Decomposition and Multi-Reward Policy Optimization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Vision-SR1: self-rewarding VLM via multi-reward policy optimization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vision-SR1 is a three-stage self-rewarding RL framework for VLMs that decomposes reasoning into explicit visual perception and language reasoning stages, then assigns separate reward signals to each via a dual-rollout mechanism. By re-prompting the same model to answer using only its generated visual description (without the image), it computes a self-visual reward without any external model or human annotation. This eliminates visual hallucinations and language shortcuts at only ~10–20% extra training compute over standard GRPO.

Problem

Standard post-training methods for VLMs supervise only the final answer, leaving intermediate visual reasoning without explicit guidance. R1-style RL worsens this by encouraging "thinking over seeing," causing models to exploit language priors (language shortcuts) and ignore visual content, leading to visual hallucinations and reward hacking. External fixes—human annotations or distillation from proprietary models—are expensive, introduce latency, and suffer from distributional shift as the policy updates.

Method

Vision-SR1 builds on GRPO and introduces a two-rollout, three-reward training loop:

  1. Reasoning decomposition: Every response follows a structured <visual reasoning>c</visual reasoning> | <think>t</think> | <answer>a</answer> format, where \(c\) must be self-contained (i.e., capture all visual information needed to answer \(q\) without the image \(i\)).

  2. First rollout (standard): \((i, q) \rightarrow (c, t, a)\). Answer reward \(r_\text{ans}(Q, a) = r_\text{acc}(Q, a) + \alpha r_\text{fmt}(s)\) is computed against ground truth.

  3. Second rollout (self-reward): \((q, c) \rightarrow (\hat{t}, \hat{a})\), image withheld. Visual reward \(r_\text{visual}(Q, c) = \mathbb{I}[\hat{a} = a^*] + \alpha r_\text{fmt}(s)\) is assigned only if the model recovers the correct answer from \(c\) alone.

  4. Multi-Reward Policy Optimization: Advantages are computed separately via group-wise z-score normalization for each reward stream: $\(A^{(i)}_\text{ans} = \frac{r_\text{ans} - \mu_\text{ans}}{\sigma_\text{ans} + \varepsilon}, \quad A^{(i)}_\text{visual} = \frac{r_\text{visual} - \mu_\text{visual}}{\sigma_\text{visual} + \varepsilon}\)$ The actor loss and KL divergence are computed separately per rollout and combined: $\(L_\text{total} = L_\text{actor} + L_\text{KL}\)$ where \(L_\text{actor} = -\frac{1}{2B}\sum_{i,t}\bigl[A^{(i)}_{\text{ans},t}\log\pi_\theta(a^{(i)}_{\text{ans},t}) + A^{(i)}_{\text{visual},t}\log\pi_\theta(a^{(i)}_{\text{visual},t})\bigr]\).

Training data is Vision-SR1-47K (~47K examples from 24 open-source benchmarks across math 30.5%, science knowledge 30%, and general visual QA 39.5%).

Key Contributions

  • Self-rewarding visual perception evaluation: the same policy model validates its own visual description via a second text-only rollout, requiring no external reward model or annotations.
  • Decoupled Multi-Reward Policy Optimization (MRPO): separate advantage computation, log-probability tracking, and KL penalties for visual and answer reward streams, preventing entanglement of heterogeneous gradients.
  • Language Shortcut Rate (LSR) metric: measures the fraction of samples where the model answers correctly despite producing a non-self-contained visual description—quantifying visual bypass behavior.
  • Near-zero GPU overhead: two-rollout training costs only ~10–20% more than standard GRPO and requires no extra GPUs (vs. external judge methods that dedicate one+ GPUs or incur API rate limits).

Results

  • Qwen2.5-VL-7B backbone: Vision-SR1 scores 40.7 MMMU-Pro and 52.2 MMMU, vs. Vision-R1 at 34.9 and 42.8 (fair 47K comparison); average across all benchmarks 52.2 vs. 44.5 for Vision-R1.
  • Qwen2.5-VL-3B backbone: average 48.8, outperforming all comparable baselines including Perception-R1 (7B, avg. 45.2) and Visionary-R1 (3B, avg. 33.9).
  • Mimo-VL-7B backbone (non-Qwen family): average improves from 44.4 (before RL) to 49.5 with Vision-SR1, demonstrating cross-architecture generalization.
  • HallusionBench: 68.3 (7B) vs. 67.4 for Vision-R1 and 65.4 for Perception-R1.
  • VisNumBench: 43.5 (7B) vs. 43.0 for Vision-R1 and 15.9 for Perception-R1.
  • Language Shortcut Rate (LSR): Vision-SR1 (7B) achieves average LSR 9.8% vs. 10.1% without self-reward; 7.5%–13.4% range across benchmarks, uniformly lower with self-reward.
  • Spatial reasoning / language-shortcut robustness (Qwen2.5-VL-7B): ViLP 52.6 vs. Vision-R1 51.3; OmniSpatial 44.2 vs. 31.1 — large gain on visual-shortcut probing.

Limitations

  • The self-visual reward is binary (\(\mathbb{I}[\hat{a}=a^*]\)), providing no partial credit for partially correct visual descriptions.
  • Two-rollout training still increases wall-clock time by ~20% (10.5h → 13h for 7B, 200 steps on 8 GPUs).
  • The self-reward signal quality is bounded by the policy model's own language reasoning ability; weak models may fail to validate even good visual descriptions.
  • LSR evaluation uses Gemini-2.5-Flash as external judge, introducing a dependency on proprietary evaluation for the proposed metric.
  • Experiments are limited to 3B and 7B models; behavior at larger scales (e.g., 72B) is not fully explored in ablations.

Relevance to Vision-Language Models

Vision-SR1 directly addresses one of the most persistent failure modes of post-trained VLMs—the collapse of visual grounding in favor of language priors—by introducing a structured reasoning decomposition that forces explicit, verifiable visual description generation. The self-rewarding mechanism is particularly significant because it scales RL supervision to the visual reasoning chain without external annotation pipelines, positioning it as a practical alternative to distillation-based approaches like Perception-R1 or Visionary-R1. The proposed LSR metric provides a principled diagnostic tool for tracking visual shortcut behavior across benchmarks, which fills a gap in standard VLM evaluation suites. For the VLM community, MRPO's decoupled advantage framework also offers a reusable design pattern for any setting where heterogeneous reward signals must be combined without entanglement.