Skip to content

SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

In-the-wild spatial reasoning benchmark challenges current VLM spatial capabilities

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatiaLab is a 1,400-item visual question-answer benchmark evaluating VLM spatial reasoning in realistic, unconstrained scenes across six principal categories and 30 task types. Tested against 25+ state-of-the-art models, the best MCQ result is 54.93% (InternVL3.5-72B) versus 87.57% for humans; open-ended performance drops a further 10–25 points. The benchmark exposes persistent, category-specific failures that prior synthetic datasets masked.

Problem

Existing spatial reasoning benchmarks rely on synthetic or LLM-generated environments, restricted task designs (often binary spatial relations or schematic navigation), and indoor/simulated imagery. These controlled setups reduce perceptual demands, produce apparent saturation, and mask failures under distribution shift — particularly for occlusion inference, scale-consistent judgments across viewpoints, and multi-step navigational planning — leaving models' real-world spatial competence unquantified.

Method

SpatiaLab constructs a benchmark through three image-collection strategies (automated web crawling, targeted online retrieval, manual snapshots), profiling images along six meta-dimensions: lighting condition, texture complexity, edge complexity, dominant spatial relation, material type, and gravity constraints. The corpus averages 21.48 objects per image, 11.88 partially visible objects, 3.23 distinct depth layers, and 2.07-link spatial reference chains; 16.71% of questions require mental rotation and 56.07% require object specificity.

Annotation follows a three-phase pipeline: (1) annotator training, (2) image-QA pairing covering proximity gradients, alignment patterns, occlusion inference, and stability prediction, (3) encoding in both 4-option MCQ and open-ended formats. A three-tier quality control procedure (semantic validity check, independent answer verification, gold-standard annotation round) yields 1,400 validated QA pairs across six categories — Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry — each decomposed into five subcategories (30 total task types, ≥200 items per category, ≥25 per subcategory).

Open-ended responses are scored by an LLM judge (Gemini-2.5-Flash) calibrated against human annotators (Cohen's \(\kappa = 0.738\) vs. majority vote; raw accuracy 0.880; Fleiss' \(\kappa = 0.774\) among annotators). Improvement strategies explored include CoT prompting, CoT with self-reflection, SFT (Qwen-VL2.5-3B fine-tuned on 40% of data with stratified sampling), and a multi-agent pipeline (SpatioXSolver).

Key Contributions

  • SpatiaLab-MCQ and SpatiaLab-OPEN: 1,400 dual-format VQA pairs at high difficulty (task complexity rated "Hi", analysis level rated "H"/"VH"), covering 30 task types drawn from real-world mixed-domain imagery.
  • Systematic evaluation of 25+ VLMs spanning proprietary, open-source (1B–321B), reasoning-focused, and spatially specialized architectures, benchmarked against human performance.
  • Fine-grained error analysis identifying concentrated failure modes in occlusion inference, scale-consistency, 3D geometry, and multi-step navigational planning.
  • Ablation of improvement strategies (CoT, CoT+self-reflection, SFT, multi-agent) showing that orientation benefits most from agentic reasoning while other categories stagnate or degrade.

Results

  • MCQ overall: InternVL3.5-72B 54.93% (best model); human baseline 87.57%; random chance 25.00%. All models span roughly 30–55%.
  • MCQ by category (InternVL3.5-72B): Relative Positioning 66.04%, Spatial Navigation 54.85%, Orientation 53.47%, 3D Geometry 50.00%, Size & Scale 49.21%, Depth & Occlusion 57.14%.
  • Open-ended overall: GPT-5-mini 40.93% (best); human baseline 64.93%. Performance drops 10–25 points relative to MCQ for all models.
  • Open-ended notable: Gemini-2.5-Pro 33.61%, GLM-4.5V-106B-MoE 25.21%; most open-source models below 25%.
  • Reasoning-focused models (MCQ): o4-mini-medium 53.21%, Gemini-2.5-Flash-Thinking 52.93% — marginal improvement over best general model.
  • Spatial specialists (MCQ): SpaceOm 41.36%, SpaceThinker-Qwen2.5VL-3B 40.64%, SpaceQwen2.5-VL-3B-Instruct 40.14% — do not outperform general models of comparable scale.
  • SFT (Qwen-VL2.5-3B): Reduced navigation and orientation errors; additive gains with CoT+SFT combination.

Limitations

  • At 1,400 items, per-subcategory counts (≥25) are small, limiting statistical power for fine-grained subcategory comparisons.
  • LLM-as-judge evaluation for open-ended tasks introduces potential systematic bias despite calibration (\(\kappa = 0.738\)).
  • Image copyright cannot be fully guaranteed despite curation efforts.
  • SFT evaluated only on the remaining 60% split of SpatiaLab itself; generalization to out-of-distribution spatial tasks is untested.
  • Spatial specialist models were trained primarily on constrained datasets; their underperformance suggests a train–test domain gap rather than a fundamental architectural limit.
  • Models with primarily open-ended training (GLM-4.5V-106B-MoE, Step-3-321B-MoE) were excluded from MCQ due to excessive refusals, creating an incomplete cross-format comparison.

Relevance to Vision-Language Models

SpatiaLab directly quantifies one of the most cited weaknesses of current VLMs — spatial reasoning — in realistic, high-clutter scenes rather than controlled synthetic environments, providing a reproducible diagnostic that exposes failures masked by prior benchmarks. The 33-percentage-point human–model gap on MCQ (87.57% vs. 54.93%) and the additional 10–25-point degradation in open-ended generation together highlight that current multimodal architectures lack robust compositional spatial representations, not merely surface-level recognition. The failure of spatially specialized fine-tuned models to outperform general-purpose frontier models suggests that perceptual grounding deficits cannot be resolved by task-specific training alone. For researchers developing VLMs for embodied AI, robotics, or autonomous navigation, SpatiaLab establishes a concrete ceiling and per-category failure taxonomy to target.