Skip to content

Grounded Reinforcement Learning for Visual Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ViGoRL (Visually Grounded Reinforcement Learning) trains VLMs to anchor each reasoning step to explicit (x, y) image coordinates, discovering that naive RL on VLMs collapses to abstract, ungrounded strategies. By bootstrapping with MCTS-generated grounded traces followed by GRPO, and extending to multi-turn RL with zoomed-in visual feedback, ViGoRL achieves state-of-the-art results across spatial reasoning, GUI grounding, and visual search benchmarks.

Problem

Standard RL applied directly to VLMs fails to induce region-level visual reasoning behaviors: base VLMs reference only ~1.44 image regions per task with minimal verification and zero backtracking, and vanilla GRPO exacerbates this by collapsing onto abstract language shortcuts that maximize reward without anchoring reasoning to visual evidence. The key gap is the absence of spatial grounding as a latent behavior that RL could amplify.

Method

ViGoRL reformulates each reasoning step as a tuple ⟨s_t, (x_t, y_t)⟩—a natural language thought paired with an image coordinate. Training proceeds in two stages:

  1. MCTS warm-start: A frozen Qwen2.5-VL-72B teacher expands nodes in a Monte Carlo tree where each node is a grounded reasoning step. From 1,500 prompts, ~30k grounded traces are generated including both direct (correct) chains and corrected (backtracking) chains. The student model (Qwen2.5-VL-3B/7B) is SFT-finetuned on linearized root-to-leaf paths.

  2. GRPO refinement: Group Relative Policy Optimization is applied with a composite reward: task correctness + a format reward that requires valid <think>/<answer> tags and valid coordinate references in all grounding calls.

Multi-turn extension: The model can issue <tool_call>{"crop": pt}</tool_call> to receive a w×w crop resized to r×r pixels at the predicted coordinate, enabling iterative zoom-in. Multi-turn GRPO adds a grammar reward (strict tag automaton) and a diversity bonus (+0.2 per sufficiently distinct coordinate, up to 4 calls).

Key Contributions

  • Diagnosis that vanilla GRPO on VLMs degrades into ungrounded reasoning, eliminating subgoal setting and backtracking behaviors
  • A grounded reasoning formalism where every CoT step is paired with an image (x, y) coordinate
  • MCTS-based warm-start data generation using a teacher model to produce exploration/backtracking traces at scale without human annotation
  • Multi-turn RL framework with tool-call-based zoom feedback for fine-grained visual inspection
  • Empirical demonstration that grounding amplifies downstream cognitive behaviors: 15× more grounded subgoals, 3× more visual verification, and unique backtracking absent in all baselines

Results

  • SAT-2 spatial reasoning: ViGoRL-3B 62.9% (+12.9 over vanilla GRPO, +16.8 over base); ViGoRL-7B 67.5%
  • BLINK: ViGoRL-3B 48.5% (+2.0 over vanilla GRPO); ViGoRL-7B 54.1%
  • RoboSpatial: ViGoRL-3B 67.1%; ViGoRL-7B 76.4%
  • ScreenSpot-V2 (GUI grounding): ViGoRL-7B 91.0% vs. 73.6% base
  • ScreenSpot-Pro: ViGoRL-3B 31.1% (+2.1 over vanilla GRPO, +21.8 over base); multi-turn variant 32.3%
  • V∗ Bench (visual search): Multi-turn ViGoRL-7B 86.4%, surpassing GPT-4o (66.0%), Sketchpad-GPT-4o (80.3%), and IVM-Enhanced GPT-4V (81.2%)
  • VisualWebArena (vision-only): ViGoRL-7B 11.2% vs. 5.5% base, outperforming ICAL-7B (8.2%) which uses additional HTML text input
  • Behavioral coding (300 SAT-2 samples): ViGoRL explores 3.5 regions vs. 1.44 (base) and 1.8 (vanilla GRPO); backtracking 0.47 vs. 0.00 in all baselines
  • Human evaluation: 72.8% of grounded predictions accurately refer to the described region; helpfulness 3.81/5 when model is correct

Limitations

  • MCTS warm-start requires a large frozen teacher (72B parameters) for node expansion, adding significant compute overhead
  • Multi-turn RL with zoom is applied only to the 7B model for V∗ Bench; ScreenSpot-Pro gains from multi-turn are modest (+1.2 pp)
  • Training data is task-specific (separate datasets per domain: spatial, web, visual search), raising questions about unified training
  • The diversity bonus in multi-turn RL is a hand-tuned heuristic (≥10px separation, up to 4 times) that may not generalize across image resolutions
  • Visual encoder still processes a globally resized image at initial steps; zoom is reactive rather than predictive

Relevance to Vision-Language Models

ViGoRL directly addresses a core weakness of modern VLMs—their tendency to treat visual input as static context rather than actively queried evidence—by making spatial grounding an explicit, learnable component of the CoT. The MCTS-to-GRPO pipeline generalizes the DeepSeek-R1-style RL-over-CoT paradigm to the visual modality, offering a training recipe applicable to any VLM base. The finding that naive RL amplifies ungrounded shortcuts is an important negative result for the growing body of work applying outcome-based RL to VLMs. The multi-turn zoom mechanism is also a concrete instantiation of agentic tool use for visual perception, connecting to broader VLM-agent research on GUI interaction and embodied navigation.