Skip to content

OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Comprehensive 3D spatial reasoning benchmark with diverse relationship categories

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniSpatial is a comprehensive spatial reasoning benchmark for VLMs grounded in cognitive psychology, covering 50 fine-grained subtasks across four dimensions: dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking, with 8.4K manually annotated QA pairs. Existing benchmarks are saturated (latest reasoning models score >90%), while all evaluated models fail dramatically on OmniSpatial, with the best model (o3) reaching only ~56% vs. human 92.6%. Two augmentation strategies—PointGraph (scene-graph cues) and SpatialCoT (novel-view chain-of-thought)—are explored to partially close the gap.

Problem

Prior spatial reasoning benchmarks (SpatialBot-Bench, EmbSpatial-Bench, VSI-Bench, etc.) target only elementary spatial relations—left/right, near/far, object counting—and are now effectively saturated: o3 and Gemini-2.5-Pro exceed 90% accuracy on them. No benchmark rigorously evaluates higher-order spatial cognition such as mental rotation, dynamic motion inference, multi-step spatial logic, or allocentric/egocentric perspective-taking, leaving a large capability blind spot for embodied AI, robotics, and autonomous driving applications.

Method

Benchmark construction: OmniSpatial collects images from web search, cognitive psychology test banks, multinational driving exams, and existing datasets (MME, HOI4D). Questions are manually authored in conversational style (avoiding rigid templates) and validated across six annotators (Krippendorff's \(\alpha = 0.84\)). The 8.4K QA pairs are split into a 1.5K test set (fully human-annotated) and a 6.9K training set augmented with SpatialViz, PhysBench, ViewSpatial, and DrivingVQA samples. Tasks span 50 subtypes across four categories: Dynamic Reasoning (27%), Perspective Taking (37%), Spatial Interaction (20%), Complex Logic (16%).

Enhancement strategies: - PointGraph: Uses Florence-2 for open-vocabulary grounding to detect objects, extract bounding boxes and centers, and assemble a JSON-format scene graph that is concatenated with the query, providing explicit geometric cues to the VLM. - SpatialCoT: Uses InstantMesh to synthesize six novel 3D viewpoints per image, composing them into a multi-view collage fed alongside the question in a chain-of-thought prompting pipeline, enabling mental imagery-style reasoning over alternative perspectives.

Key Contributions

  • Taxonomy of visual-spatial reasoning into four cognitively grounded dimensions with 50 fine-grained subcategories, exposing challenges beyond existing saturated benchmarks.
  • OmniSpatial dataset: 8.4K manually annotated QA pairs across diverse scenes, resolutions, and countries; includes a train/test split for both evaluation and fine-tuning.
  • Systematic evaluation of 36 VLMs spanning proprietary, reasoning, open-source, and spatially specialized model families under a unified zero-shot protocol.
  • PointGraph and SpatialCoT augmentation strategies demonstrating that explicit scene-graph cues and novel-view synthesis improve spatial reasoning, with PointGraph yielding up to +2.91% average accuracy on smaller models.

Results

  • Best model overall: o3 at 56.33% average accuracy; human performance is 92.63% — a gap of ~36 points.
  • Top proprietary: Gemini-2.5-Pro at 55.19%, Gemini-2.5-Flash at 53.16%; GPT-4.1 at 51.78%.
  • Top open-source: InternVL3-78B at 49.33%, Qwen-VL2.5-72B at 47.85%.
  • Specialized spatial models underperform general VLMs of comparable size; best is SoFar-Qwen2.5VL-3B at 45.14%.
  • Hardest subcategories: Pattern Recognition (~27–38% for top models) and Geospatial Localization/Strategy (~21–43%), suggesting abstract spatial logic is the main bottleneck.
  • PointGraph improves GPT-4.1-mini by +1.63%, Gemini-2.5-Flash by +1.76%, Qwen-VL2.5-3B by +2.91%.
  • Textual CoT (zero-shot or manual) provides marginal gains (+0.06% to +0.95% on larger models) and can hurt smaller models (−0.81% to −1.38%).
  • All evaluated models scored >90% on prior benchmarks (SpatialBot-Bench, EmbSpatial), confirming saturation.

Limitations

  • Test set is 1.5K QA pairs; per-subcategory sample counts may be small for some of the 50 fine-grained types.
  • SpatialCoT relies on InstantMesh NVS quality; reconstruction artifacts in complex or texture-less scenes may degrade rather than help reasoning.
  • PointGraph depends on open-vocabulary detector accuracy (Florence-2); detection errors propagate into the scene graph.
  • Dynamic reasoning tasks use static or sparsely sampled frames as a proxy for true video-temporal reasoning, limiting ecological validity for motion-heavy tasks.
  • Benchmark is English-language-centric despite cross-country image collection; multilingual evaluation is not addressed.
  • No fine-tuning experiments on the full 6.9K training split are reported in the main paper to quantify training benefit.

Relevance to Vision-Language Models

OmniSpatial directly exposes a critical capability frontier for VLMs: comprehensive spatial cognition, which underlies embodied AI, robotics, and autonomous driving applications. By demonstrating that state-of-the-art reasoning models (o3, Gemini-2.5-Pro) plateau near 55% on tasks humans solve at 93%, it provides a rigorous diagnostic separating genuine spatial understanding from benchmark overfitting. The two proposed augmentation strategies—PointGraph and SpatialCoT—are lightweight, model-agnostic wrappers applicable to any VLM, offering practical paths for improving spatial grounding without retraining. This benchmark will serve as a standard testbed for the next generation of spatially and physically aware multimodal models.