Skip to content

Spatial Understanding from Videos: Structured Prompts Meet Simulation Data

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a two-pronged framework—SpatialMind (structured CoT prompting) and ScanForgeQA (large-scale synthetic QA dataset)—to improve 3D spatial reasoning in pre-trained VLMs operating on scanning videos, without modifying model architecture. SpatialMind decomposes scenes into global 3D maps and question-type-specific reasoning chains; ScanForgeQA provides 925K QA pairs from 34K simulated single-room scenes. Combined, they yield consistent +6–8% gains across multiple spatial benchmarks.

Problem

Existing VLMs lack robust 3D spatial reasoning when input is limited to scanning videos (no depth sensors or point clouds). Two root causes: (1) spatial uncertainty—inferring 3D structure from 2D multi-frame video requires multi-step cross-frame reasoning; (2) data scarcity—real-world 3D QA datasets (e.g., ScanNet-derived) are small-scale (<300K QA pairs) and non-scalable, insufficient for teaching fine-grained spatial commonsense.

Method

SpatialMind (prompting, training-free): - Scene Decomposition: (a) Local Modeling — per-frame detection of candidate objects with VLM-estimated local 3D positions; (b) Coordinate Mapping — accumulates relative rotations/translations between adjacent frames to build a global 3D map; (c) Cognition Generation — converts the 3D map to a 2D grid or natural language spatial descriptions (e.g., "monitor: 1m left of reference"). - Question Decomposition: classifies questions into types (relative distance, direction, contact, etc.) and routes each to a GPT-4o-designed, human-verified step-by-step reasoning plan. - Textual position descriptions outperform both 3D map and 2D grid formats across all tested VLMs.

ScanForgeQA (fine-tuning dataset): - Scene construction: 34,116 single-room scenes from 3D-FRONT (disassembled multi-room scans) + 160 LLM-generated scenes via HoloDeck/Objaverse. - Scan creation: Unity engine with orbit (72 frames, 360°) and navigation (72 frames, start/end rotations + path traversal) scanning strategies. - QA generation: three categories—attribute estimation, spatial reasoning, hypothesis analysis—derived from ground-truth 3D annotations; 925K QA pairs total.

Key Contributions

  • SpatialMind: architecture-agnostic structured CoT prompting that decomposes spatial scenes and questions into interpretable reasoning steps across frames.
  • ScanForgeQA: scalable synthetic dataset (34K scenes, 103K video scans, 925K QA pairs) built via automated pipeline, an order of magnitude larger than prior real-scan datasets.
  • Empirical finding that textual spatial descriptions are more VLM-interpretable than structured 3D maps or 2D grids.
  • Analysis showing prompting and fine-tuning are complementary: larger models benefit more from prompting; smaller models benefit more from fine-tuning.
  • Mixed fine-tuning strategy (5–10% general video data + ScanForgeQA) that recovers general video understanding regression.

Results

VSI-Bench (primary benchmark, 8 spatial sub-tasks): - Gemini-1.5 Pro: 45.4 → 52.8 (+7.4%) with SpatialMind - GPT-4o: 34.0 → 40.8 (+6.8%) with SpatialMind - InternVL2-8B: 34.6 → 42.7 (+8.1%) with both; 38.9 (+4.3%) prompting only, 41.5 (+6.9%) fine-tuning only - InternVL2-40B: 36.0 → 44.5 (+8.5%) with both - Qwen2.5-VL-7B: 37.2 → 43.9 (+6.7%) with both - Qwen2.5-VL-72B: 39.2 → 47.1 (+7.9%) with both

Generalization benchmarks (Qwen2.5-VL-7B / 72B, +Both): - OpenEQA: 50.1/3.1 → 58.6/3.5 (Acc/Score) / 53.8 → 60.4 - ScanQA BLEU-1: 32.5 → 37.9 / 35.4 → 44.1 - SQA3D EM-1: 17.2 → 24.5 / 34.8 → 46.3

ScanForgeQA fine-tuning on VSI-Bench outperforms fine-tuning on SQA3D (38.8) and ScanQA (38.5) baselines (Qwen2.5-VL-7B avg 43.3 vs. 38.8/38.5).

Limitations

  • Relies on GPT-4o calls for scene parsing and question-type routing, introducing API cost and latency.
  • Fine-tuning on ScanForgeQA causes a marginal drop on Video-MME (event-comprehension tasks), requiring mixed-data mitigation.
  • All simulated scenes are single-room indoor environments; generalization to outdoor or multi-room scenes is untested.
  • Coordinate mapping via VLM-estimated relative pose is noisy; no quantitative analysis of pose estimation error vs. downstream QA error.
  • Architecture is unchanged—the framework cannot correct for fundamental VLM perceptual failures (e.g., object hallucination under heavy occlusion).

Relevance to Vision-Language Models

This work directly targets a known weakness of VLMs—3D spatial reasoning from purely visual input—and does so without architectural changes, making it immediately applicable to any existing VLM. The finding that textual spatial descriptions outperform structured 3D formats (maps, grids) is an actionable insight about VLM inductive biases, relevant to anyone designing spatial prompts. ScanForgeQA fills a critical data gap for fine-tuning spatial reasoning in VLMs and the scalable simulation pipeline is a reusable template. The complementarity of prompting (more effective for large models) vs. fine-tuning (more effective for small models) is a useful practical heuristic for the VLM adaptation literature.