Skip to content

Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ViLaSR introduces "drawing to reason in space," a paradigm where LVLMs iteratively annotate bounding boxes and draw auxiliary lines directly on images as part of their reasoning chain, avoiding reliance on external perception tools. A three-stage training framework—cold-start SFT, reflective rejection sampling, and GRPO-based RL—cultivates this visual manipulation capability. The model achieves 18.4% average improvement over strong baselines across five spatial reasoning benchmarks.

Problem

Existing LVLMs perform spatial reasoning purely in text, but spatial details (object positions, trajectories, perspective relationships) are lossy when encoded into textual space. Tool-integrated reasoning approaches that invoke external perception APIs (grounding, OCR) are constrained by those tools' fixed capabilities and produce fragmented, non-reflective reasoning chains. Neither approach supports the iterative self-correction that hard spatial tasks (maze navigation, multi-view, video) demand.

Method

Reasoning paradigm. At each step \(t\), the model generates natural-language reasoning \(r_t\) and a set of drawing operations \(e_t = \{(k, p, l)\}\) (bounding-box \(T_\text{box}\) or auxiliary-line \(T_\text{line}\)), executes them to produce annotated images \(o_t\), and feeds \((r_t, e_t, o_t)\) back into the next step. The model reasons over its own edited images, not just the originals.

Three-stage training on Qwen2.5-VL-7B: 1. Cold-start SFT. Qwen2.5-72B-VL generates synthetic reasoning paths following the paradigm; paths passing rule-based correctness + format checks form dataset \(D_\text{cold}\); standard cross-entropy fine-tuning. 2. Reflective rejection sampling. The cold-started model rolls out paths on \(D_\text{reflect}\); only paths that are correct, well-formatted, and exhibit reflection (same semantic label reused with different operation parameters across steps) are kept for continued SFT. This doubles reflection frequency before RL. 3. Reinforcement learning (GRPO, no KL penalty). Reward \(S = \mathbf{1}[S_\text{correct}>\beta]\cdot S_\text{correct} + S_\text{format}\); correctness uses exact match for multiple-choice and Mean Relative Accuracy (MRA across thresholds \(C=\{0.50,\ldots,0.95\}\)) for numerical questions. Rollout early-termination kills circular or tool-free trajectories.

Key Contributions

  • "Drawing to reason in space" paradigm: native visual manipulation (bbox + auxiliary lines) as first-class reasoning tokens, no external tool dependency.
  • Reflective rejection sampling stage that explicitly selects for self-correction behavior before RL, addressing the cold-start model's near-zero reflection rate.
  • GRPO-based RL with MRA rewards enabling fine-grained learning for numerical spatial quantities (distances, sizes).
  • State-of-the-art on five spatial benchmarks spanning image, video, and multi-view reasoning using only a 7B backbone.

Results

  • MAZE (image sequential planning): 98.2% vs. Qwen2.5-VL-7B baseline 33.7% (+64.5 pp); beats GPT-4o (48.8%) and OpenAI o3 (79.0%).
  • SpatialEval-Real: 63.9% (+5.4 pp over baseline 58.5%).
  • VSI-Bench (video): 45.4% (+12.7 pp over baseline 32.7%); beats LLaVA-OneVision-72B (40.2%) and Gemini-2.0-Flash (45.4†).
  • SPAR-Bench (multi-view): 37.6% (+5.9 pp).
  • MMSI-Bench (multi-view): 30.2% (+3.3 pp); beats LLaVA-OneVision-72B (28.4%).
  • Average improvement: 18.4% over Qwen2.5-VL-7B without reasoning.
  • Inference-time scaling (VSI-Bench pass@8): 0.461 for ViLaSR vs. 0.328 for cold-start-only model; RL narrows pass@1/pass@8 gap (0.408 vs. 0.461 → gap of 0.053 vs. 0.100 without Stage 3).
  • Removing reflective rejection sampling (Stage 2) causes 96.5% drop in reflection exhibition and 85% drop in \(T_\text{box}\) usage, disproportionately hurting precise measurement subtasks.

Limitations

  • Drawing operations cover only bounding boxes and auxiliary lines; 3D spatial relationships and more complex geometric manipulations are not supported.
  • Training data restricted to multiple-choice and numerical questions; free-form spatial descriptions (e.g., motion trajectory narration) are excluded from training.
  • Video inputs are uniformly downsampled to 16 frames at 256Ă—28Ă—28 resolution, potentially losing fine-grained temporal detail.
  • RL training response length decreased from ~2,500 to ~1,800 tokens, suggesting the model may not develop sufficiently long reasoning chains for the hardest tasks; authors attribute this partly to limited training data diversity.
  • Evaluation does not include open-ended generation benchmarks, so generalization of the drawing paradigm to unconstrained spatial description tasks is untested.

Relevance to Vision-Language Models

ViLaSR directly challenges the dominant text-centric LVLM reasoning paradigm by demonstrating that native visual manipulation—rather than language translation of spatial content—is necessary for precise geometric and temporal tracking. The three-stage curriculum (cold-start → reflective sampling → RL) provides a reproducible recipe for instilling emergent visual self-correction in 7B-scale models, which is directly relevant to the line of work applying slow-thinking/GRPO training (DeepSeek-R1 style) to multimodal models. The result that open-source models without this training show negative returns from added reasoning steps underscores a critical capability gap, positioning ViLaSR's framework as a prerequisite for reliable spatial VLM deployment in robotics and AR settings. The work also establishes that tool-free, intrinsic drawing operations can outperform external-API-augmented reasoning, broadening the design space for future VLM architectures.