Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

🕒 Published (v1): 2026-07-02 17:53 UTC · Source: Arxiv · link

Why this paper was selected

Durrett lab; RL trains visual self-reflection/CoT correction — key for VLM agent reasoning chains

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VRRL (Visual Reflection RL) is a two-stage SFT→RL training framework that teaches large vision-language models to iteratively correct predictions using visual feedback from the environment. Two novel RL mechanisms—Random Turn Masking and Buffered Roll-In—explicitly train error recovery from diverse intermediate failure states. The method substantially improves out-of-distribution accuracy on visual grounding and spatial navigation over standard RL and reflection-oriented fine-tuning baselines.

Problem

LVLMs exhibit a modality gap during self-reflection: they fail to attend to relevant visual tokens and cannot translate environment-provided visual feedback into grounded corrective behavior, particularly on OOD inputs that diverge from pre-training distributions. Prompting-based and textual-CoT-only reflection approaches either repeat previous predictions or fail to correct mistakes, and SFT on recovery trajectories teaches format but not robust error-correcting behavior.

Method

Stage 1 — SFT: A dataset \(D_\text{SFT}\) of single-turn (correct, immediate) and multi-turn (synthetic error-then-correction) trajectories is used to initialize the policy with the interaction format and basic visual error correction via standard cross-entropy loss on assistant turns \(a_1,\ldots,a_T\) (the erroneous \(a_0\) is excluded from the loss).

Stage 2 — GRPO with two mechanisms:

Random Turn Masking (RTM): For a rollout of length \(T\), a start index \(k \sim \text{Unif}(1, T)\) is sampled and policy-gradient updates are computed only on the suffix \([k, T]\): $\(\nabla J_\text{RTM}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \mathbb{E}_{k \sim \mathcal{U}(1,T)} \left[\sum_{t=k}^{T} \nabla \log \pi_\theta(a_t|s_t)\hat{A}_t\right]\)$ This trains recovery from arbitrary intermediate states without learning to produce the preceding mistakes.

Buffered Roll-In: A replay buffer \(B\) stores prefixes of failed trajectories (the state just before a wrong termination). During training, sampled prefixes are extended by \(G\) suffix completions; GRPO is applied only to the generated suffixes, maintaining a supply of difficult recovery scenarios as the on-policy failure rate drops.

The combined objective is \(J_\text{Total}(\theta) = \rho J_\text{RTM}(\theta) + (1-\rho) J_\text{Buff}(\theta)\), creating a self-paced curriculum.

Reward combines: format validity \(r_\text{fmt}\), binary outcome reward \(R_\text{answer}\), and a task-specific improvement-based reflection reward \(R_\text{refl}\), with \(R = \max(R_\text{answer}, R_\text{refl})\) when format is valid.

Tasks: (1) Visual grounding on synthetic table/chart images, where the model predicts pixel coordinates \(p=(x,y)\) and receives a \(200{\times}200\) crop centered at the prediction with a red marker as feedback; OOD splits include larger tables, cell queries, bar charts, and scatter plots. (2) Spatial navigation on FrozenLake grid maps, where the model predicts full paths and receives a rendered overlay of the proposed path; OOD splits use larger \(6{\times}6\) and \(7{\times}7\) grids.

Key Contributions

• Demonstrates that visually grounded self-reflection (driven by rendered visual feedback) is a more effective error-correction mechanism than textual CoT reflection or prompting for OOD robustness in LVLMs.
• Introduces Random Turn Masking, which reweights GRPO gradients to train recovery from arbitrary intermediate states rather than just full-trajectory optimization.
• Introduces Buffered Roll-In, which repopulates training with historically difficult failure prefixes to sustain error-recovery signal as on-policy failure rates decrease.
• Shows that combining both mechanisms in VRRL induces more efficient and selective reflection behavior (improving over turns, not just single-shot accuracy).

Results

Visual Grounding (Table 1): - Qwen2.5-VL-3B: VRRL OOD avg 45.7% vs. Multi-SFT→GRPO 40.0% vs. Single-SFT→GRPO 30.1%; Cell Query 20.3% vs. 13.5%; Scatter Plot 40.3% vs. 37.2%. - Qwen2.5-VL-7B: VRRL OOD avg 78.4% vs. Multi-SFT→GRPO 73.2%; Cell Query 77.3% vs. 68.4%; Scatter Plot 57.0% vs. 46.8%. - VL-Rethinker-32B (textual reflection trained on diverse tasks) achieves 22.1% OOD avg on the 3B scale, well below VRRL's 45.7%.

Spatial Navigation (Table 2): - Qwen2.5-VL-3B: VRRL OOD avg 39.2% vs. Reflection Tuning 31.0% vs. Multi-SFT→GRPO 26.0%. - Qwen3-VL-4B: VRRL OOD avg 52.2% vs. Reflection Tuning 49.6% vs. Multi-SFT→GRPO 36.0%. - VRRL improves OOD over Multi-SFT baseline by 13.3% (3B) and 11.4% (4B), outperforming the second-best baseline by 9.8% on average across both models.

Reflection efficiency (Table 3): On Qwen2.5-VL-3B, VRRL achieves \(\Delta_\text{ref}=+6.0\) ID and \(+23.0\) OOD improvement-over-turns vs. Multi-SFT→GRPO's \(+3.4\) and \(+7.4\), using a similar average turn count.

Limitations

• Evaluated on only two task families (coordinate-level visual grounding and grid-based navigation); generalization to richer open-domain tasks is unverified.
• Requires an environment that can render visual feedback at each turn; the framework is not directly applicable to tasks without a structured feedback-rendering mechanism.
• Standard GRPO applied to multi-turn models (Multi-SFT→GRPO) suppresses reflection in spatial navigation, indicating the reflection reward and masking components are critical but their relative contributions are task-dependent.
• VRRL slightly underperforms Single-SFT→GRPO on in-distribution spatial navigation for Qwen2.5-VL-3B (83.9% vs. 85.9%), suggesting a small in-distribution trade-off.
• Synthetic training data (15K/6K for grounding, 4K/2K for navigation) may limit scale, and the replay buffer strategy may not scale trivially to tasks with exponentially large failure prefix spaces.

Relevance to Vision-Language Models

This paper directly addresses a known failure mode of LVLMs—the modality gap in self-reflection—by designing RL training objectives that force the model to use rendered visual feedback rather than relying solely on textual reasoning. The VRRL framework is a principled extension of GRPO to multi-turn visual feedback loops, connecting to the broader line of work on post-training for reasoning (GRPO, RLVR) while extending it to the multimodal, sequential-decision setting. The OOD generalization results are practically significant: they show that training on one visual domain (small synthetic tables) can transfer to structurally different domains (bar charts, scatter plots) when the model learns to close the loop through visual feedback. For VLM researchers, this establishes visually grounded reflection as a distinct and trainable capability that complements but does not reduce to longer textual CoT scaling.