Skip to content

SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatialReasoner is an LVLM that tackles 3D spatial reasoning by grounding multi-stage inference (perception → computation → reasoning) in explicit 3D representations (object locations and orientations in calibrated camera space), rather than relying on implicit natural language descriptions. A two-stage SFT→RL post-training pipeline achieves state-of-the-art results on 3DSRBench (60.3%) while generalizing better to novel question types unseen during training.

Problem

Existing 3D-aware VLMs (SpatialRGPT, SpatialLLM) and proprietary models (Gemini 2.0, GPT-4o) perform 3D spatial reasoning implicitly through natural language, which is both imprecise and non-compositional. Long chain-of-thought reasoning fails on questions trivial to humans because models lack reliable 3D computation: they can decompose problems but cannot accurately execute geometric operations (distance, angle, relative orientation estimation).

Method

SpatialReasoner builds on Qwen2.5-VL-7B and introduces two coupled components:

Explicit 3D representations: Objects are localized and oriented as 3D vectors in a calibrated camera space (z-axis aligned to world vertical, origin near ground plane). This simplifies height comparisons to z-coordinate comparisons and reduces many 3D relationships to 2D. These representations serve as a typed intermediate interface shared across perception, computation, and reasoning stages.

Two-stage post-training: - Stage I – SFT: Fine-tuning on synthetic 3D-annotated QA data in three flavors: Basic3D-QA (detection + pose), SR-QA (spatial relationship VQA), and SR-CoT (chain-of-thought with explicit 3D steps interleaved). Pseudo-annotations are generated from OpenImages via segmentation, metric depth, and pose estimation models, then aggressively filtered and optionally human-verified (1.2K verified subset). - Stage II – RL: GRPO optimization with a composite reward: answer accuracy (MMBench-style multiple-choice), format reward, and optional 3D-aware process rewards (checking for spatial terms like "distance", "orientation" in reasoning traces). KL divergence regularization is ablated away as it causes training collapse.

Key Contributions

  • Explicit 3D representation interface (calibrated camera space) enabling interpretable, compositional multi-stage spatial reasoning in an LVLM.
  • Two-stage SFT+RL post-training recipe that balances perceptual grounding (SFT) with adaptive generalization (RL).
  • Empirical failure mode analysis: by inspecting intermediate 3D outputs on 300 manually verified questions, the paper quantifies that 3D perception accuracy lags behind 3D reasoning accuracy, pinpointing perception as the primary error source.
  • Ablation study establishing that RL generalizes better OOD, SFT tolerates noisy data better, repeated SFT (SFT+SFT) hurts performance, and removing KL from GRPO improves training stability.

Results

  • 3DSRBench (2,100 Qs): SpatialReasoner achieves 60.3% mean accuracy, outperforming Gemini 2.0 Flash (49.8%, +10.5 pp), Gemini 2.0 Flash Thinking (51.1%, +9.2 pp), Claude 3.5 Sonnet (48.2%), and best open-source specialist SpatialLLM (44.8%).
  • Orientation sub-category: 55.2% vs. 43.1% (Qwen2.5-VL-72B) and 35.8% (Gemini 2.0 Thinking).
  • Multi-object sub-category: 51.8%, +8.2 pp over second-best model.
  • CVBench-3D (Depth): 87.3% vs. Qwen2.5-VL-7B baseline 82.5%.
  • GQA (Compare): 81.1% vs. 71.5% (Qwen2.5-VL-7B).
  • OOD generalization (multi-object questions withheld from training, Table 3): SpatialReasoner-Zero (RL-only) achieves 46.6% vs. SpatialReasoner-SFT 40.0%, demonstrating RL's superior zero-shot transfer.
  • Ablation – explicit 3D reps: removing them from SFT drops mean from 58.3% to 51.9% on 3DSRBench.
  • Ablation – SFT+SFT vs. SFT+RL: second SFT round degrades to 54.7%; RL after SFT reaches 60.3%.

Limitations

  • 3D pseudo-annotations derived from visual foundation models contain factual errors (missing objects, inaccurate poses); human verification is expensive and limited to 1.2K samples.
  • RL favors high-quality verified data but high-quality 3D data is scarce, limiting scalability of the RL stage compared to SFT.
  • CVBench-3D distance questions show a performance drop, attributed to the benchmark's reliance on 2D shortcuts that SpatialReasoner deliberately avoids—raising questions about benchmark alignment.
  • RL slightly degrades intermediate perception and reasoning accuracy (as measured on manually verified 3D answers), because the reward signal targets final answers and format, not intermediate 3D outputs.
  • Requires camera calibration (extrinsics) to define the calibrated 3D space; applicability to uncalibrated wild images is not fully addressed.

Relevance to Vision-Language Models

SpatialReasoner directly advances the line of work on spatially-grounded VLMs (SpatialVLM, SpatialRGPT, SpatialLLM) by demonstrating that explicit structured representations—rather than free-form language descriptions—are critical for reliable 3D computation inside an LVLM. The SFT→RL training recipe and the finding that "SFT memorizes, RL generalizes" have broad implications for post-training strategies in VLMs beyond 3D tasks. The interpretability analysis, enabled by inspecting intermediate 3D outputs, provides a concrete methodology for diagnosing where VLMs fail in multi-step visual reasoning, a recurring challenge across VLM research.