Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; unified spatial reasoning benchmark spanning robotics, AR, and navigation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Spatial-DISE introduces a cognitively grounded 2×2 benchmark taxonomy (Intrinsic/Extrinsic × Static/Dynamic) for evaluating spatial reasoning in VLMs, addressing the systematic neglect of dynamic, multi-step mental transformation tasks. The benchmark provides 559 evaluation VQA pairs and 12K+ training pairs synthesized via Blender, revealing that 32 evaluated VLMs average only 28.4% accuracy against a 76.8% human baseline.

Problem

Existing spatial reasoning benchmarks share three critical gaps: (1) no unified cognitive framework, leading to fragmented and static-biased evaluation; (2) near-complete absence of intrinsic-dynamic tasks (mental rotation, folding, punch-and-unfold) that are foundational to human spatial cognition; (3) insufficient scale in the few benchmarks that do address dynamic reasoning, preventing robust model analysis or training data generation.

Method

The DISE taxonomy partitions spatial reasoning into four quadrants along two cognitive dimensions: - Intrinsic vs. Extrinsic: whether reasoning concerns a single object's internal structure or inter-object spatial relations. - Static vs. Dynamic: whether mental transformation is required.

Ten tasks map to these quadrants: I-S (2D/3D Shape Finding), I-D (2D/3D Rotation, 2D/3D Folding, Fold-and-Punch), E-S (3D Projection), E-D (2D/3D Combination).

Data is produced by a three-stage pipeline: 1. Real-world collection: 1,180 VQA pairs from psychometric and aptitude tests serve as templates. 2. Scalable synthetic generation via Blender: Each instance is seeded by a hashed question_id for reproducibility. The pipeline programmatically generates core 3D assets, renders question/answer images from optimal camera angles, and constructs distractors through four tiered strategies — geometric variations, pattern/orientation errors, incorrect views, and component replacement — under controlled lighting and camera parameters. 3. Human quality control: 559 benchmark pairs pass three criteria: solution uniqueness, rendering/question clarity, and non-redundancy. Over 12,000 pairs form the training split.

Evaluation uses VLMEvalKit with exact-match accuracy; human baseline uses 54 participants (ages 15–55), matrix-sampling design, and IRT cross-validation over 1,679 responses.

Key Contributions

• DISE taxonomy: a cognitively grounded 2×2 framework that classifies any spatial task into one of four quadrants, enabling structured diagnosis of model weaknesses.
• Blender-based verifiable generation pipeline: fully automated, seeded, and reproducible; reusable by the community for generating dynamic 3D spatial reasoning data at scale.
• Spatial-DISE Bench: 559 VQA pairs across all 10 tasks and four DISE quadrants for benchmarking.
• Spatial-DISE-12K: 12K+ verifiable VQA pairs across five 3D tasks for training/fine-tuning.
• Comprehensive evaluation of 32 VLMs spanning proprietary, open-source, reasoning-augmented, and spatially specialized models, establishing current capability boundaries.

Results

• Average accuracy across all 32 models: 28.4% (random chance 24.8%, human baseline 76.8%).
• Best overall model: Doubao1.5-VL-thinking at 42.0%.
• Fold-and-Punch (multi-step I-D): mean across all models 25.4%; best model (Doubao-1.5-thinking) 30.8% — near random chance.
• SFT on Spatial-DISE-12K: Qwen2.5-VL-7B-sft reaches 47.0% (+20.9 pp over base); SpaceOM-sft reaches 41.3% (+15.4 pp).
• Counterintuitive finding: Gemini2.0-Flash scores 38.3% on dynamic tasks vs. 23.6% on static, suggesting different failure modes between quadrants.
• Doubao-1.5-thinking exceeds human performance on Extrinsic-Dynamic questions but falls far short overall.
• Post-training (RL/SFT) improves scores but all models remain far below human baseline.

Limitations

• Spatial-DISE Bench is small (559 pairs), making per-task confidence intervals wide (many tasks have \(n < 100\)).
• Spatial-DISE-12K covers only five 3D tasks; I-S and E-S dynamic categories lack training data.
• Post-training gains are modest in absolute terms, and the gap to human competence remains large even after fine-tuning on in-distribution data.
• Error analysis (referenced but truncated in the text) is not fully elaborated in the provided excerpt, limiting insight into failure mode taxonomy.
• The benchmark focuses on multiple-choice VQA; open-ended or generative spatial tasks are not evaluated.

Relevance to Vision-Language Models

Spatial-DISE directly addresses a diagnostic blind spot in VLM evaluation: current benchmarks systematically undertest the cognitive processes (mental rotation, multi-step mental simulation) that distinguish genuine spatial understanding from pattern matching. The finding that 32 VLMs — including GPT-5, o4-mini, and Gemini 2.5 Flash — average only 3.6 pp above random chance underscores that scaling and RLHF-based alignment alone do not resolve spatial cognition deficits. The 12K training split and open pipeline make this benchmark directly usable for future VLM post-training targeting spatial intelligence, and the DISE taxonomy provides a reusable diagnostic lens for tracking progress across model generations.