Skip to content

SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence

🕒 Published (v1): 2025-05-22 17:59 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatialScore is a comprehensive benchmark of 5,025 manually verified QA samples spanning 30 spatial reasoning tasks across 10 categories, used to evaluate 49 MLLMs. The authors complement it with SpatialCorpus (331K fine-tuning samples) and SpatialAgent, a multi-agent framework with 12 specialized tools operating under Plan-Execute or ReAct paradigms. The best model (Gemini-3-Pro, 60.12%) still trails human performance (86.60%) by a 26+ point margin.

Problem

Existing spatial intelligence evaluations of MLLMs are fragmented: they rely on oversimplified tasks (coarse yes/no spatial queries), single-modality inputs, narrow skill coverage, and inconsistent QA formats, making it impossible to holistically measure spatial understanding including rigorous 3D geometry perception (camera pose, depth, homography).

Method

Benchmark (SpatialScore): 3D annotations from ScanNet++, Omni3D, WildRGB-D, PointOdyssey, and CA-1M are repurposed into QA pairs via template-based generation and LLM rewriting (DeepSeek-v3). These are merged with 23 existing spatial datasets (63,857 candidates), filtered by GPT-OSS-120B to remove visually independent questions, then manually verified by five annotators to yield 5,025 samples spanning MCQ, Yes/No, and open-ended formats.

Training corpus (SpatialCorpus): 331K multimodal QA pairs covering real-world and simulated scenes (273K real/simulation + 58K rendered), used for supervised fine-tuning of models such as Qwen3-VL.

SpatialAgent: A training-free multi-agent system formalized as: $\(r = A(q, v; \Phi; T)\)$ where \(T = \{t_1, t_2, \ldots, t_n\}\) is a toolbox of 12 specialized spatial perception tools (depth estimator, camera pose estimator, motion estimator, etc.). It operates under two reasoning paradigms: - Plan-Execute: hierarchically decomposes tasks into sub-tasks with sequential tool invocation. - ReAct: interleaved reasoning-and-action loop that refines decisions via context-aware tool interactions.

Key Contributions

  • SpatialScore: 5,025-sample benchmark across 30 tasks, 10 categories, covering real/simulated/AIGC data, image/video/sequence inputs, and MCQ/Yes-No/open-ended formats — largest and most diverse spatial benchmark to date.
  • Systematic evaluation of 49 MLLMs across scale groups (1B–235B) and proprietary APIs (GPT-5, Gemini, Claude).
  • SpatialCorpus: 331K training samples enabling supervised fine-tuning for spatial reasoning (demonstrated on Qwen3-VL).
  • SpatialAgent: training-free multi-agent framework with 12 tools and dual reasoning modes (Plan-Execute / ReAct).

Results

  • Human-level performance on SpatialScore: 86.60% overall.
  • Best proprietary model: Gemini-3-Pro (60.12%), followed by GPT-5 (58.13%) and Gemini-2.5-Pro (56.37%).
  • Best open-source model: Qwen3-VL-235B-A22B (56.63%).
  • Gap between best model and human: 26.48 points.
  • Most models spatially fine-tuned (SpaceThinker, SpatialThinker, SpatialReasoner) show only marginal gains or regress vs. base models; VST is a notable exception with significant gains.
  • Models near-human on mental animation and object localization but severely underperform on view reasoning, camera pose/motion, and real-world 3D perception.
  • Blind text-only GPT-5 scores 30.62%, barely above random chance (28.29%), confirming visual grounding is necessary.

Limitations

  • SpatialAgent toolbox (12 tools) may not cover all spatial perception subtasks; tool selection quality depends on the backbone MLLM's reasoning.
  • SpatialCorpus fine-tuning gains demonstrated primarily on Qwen3-VL; generalizability to other architectures unquantified in the text provided.
  • Most spatially specialized fine-tuned models fail to generalize, suggesting current training paradigms are insufficient — but the paper does not resolve why.
  • Dataset curation relies on GPT-OSS-120B for filtering, introducing potential biases from that model's own spatial blind spots.
  • Coverage of dynamic 3D scenes and long-horizon temporal reasoning remains thin.

Relevance to Harnesses / Meta-Harnesses

SpatialAgent is a direct instance of a domain-specialized tool-orchestration harness: a controller MLLM dispatches to a curated toolbox of 12 perception specialists under either a Plan-Execute or ReAct meta-loop, without retraining the backbone. The dual-paradigm design (hierarchical decomposition vs. iterative refinement) maps precisely onto the architectural choices explored in agentic meta-harnesses — Plan-Execute mirrors static DAG-style orchestration while ReAct mirrors dynamic, feedback-driven loops. The benchmark itself functions as a harness-evaluation scaffold, systematically probing 49 models under standardized conditions across heterogeneous task types and modalities. For researchers building meta-harnesses, this work demonstrates that training-free tool orchestration can compensate for backbone deficiencies in specialized perception domains, and that benchmark comprehensiveness (task diversity + modality coverage) is essential for diagnosing harness failure modes.