Skip to content

SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; SpinBench: perspective-taking diagnostic for spatial reasoning in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpinBench is a cognitively grounded diagnostic benchmark evaluating spatial reasoning in VLMs through the lens of perspective taking, decomposed into seven progressive task categories. Evaluating 43 models reveals systematic failures: near-chance performance on mental rotation and perspective taking, strong egocentric bias, and severe inconsistency under logically equivalent reformulations. Humans achieve 91.2% accuracy, and human response time correlates strongly with VLM accuracy, validating the benchmark's cognitive fidelity.

Problem

Existing spatial reasoning benchmarks either entangle spatial reasoning with high-level planning or lack controlled variation across reference frames, premise structures, and linguistic reformulations. As a result, it remains unclear whether VLMs genuinely perform geometry-aware spatial inference or rely on shallow linguistic pattern matching. No prior benchmark progressively scaffolds from atomic perceptual skills to compositional perspective taking, nor systematically isolates failure modes across rotation, translation, and viewpoint change.

Method

SpinBench structures evaluation around perspective taking as the apex integrative skill, decomposed into seven ordered task categories: (1) identity matching, (2) object-relation grounding, (3) dynamic translation, (4) dynamic rotation, (5) canonical view selection, (6) mental rotation, and (7) perspective taking. All tasks are constrained to horizontal 2D orbital viewpoint changes to eliminate vertical confounds.

The benchmark spans four visual domains—synthetic Infinigen/YCB tabletop scenes, ABO household 3D objects, Multi-View Car Dataset, and Stereo Face Database—yielding 2,739 samples across 51 tasks. Controlled variation axes include: (i) frame-of-reference (egocentric vs. allocentric, tested explicitly via face rotation tasks), (ii) premise-based conditions (visual inference vs. linguistically provided spatial relation) to disentangle visual grounding from geometric reasoning, (iii) symmetrical augmentation (flipping question and answer), and (iv) syntactic augmentation (rephrasing questions while preserving semantics). Performance is measured via raw accuracy, Cohen's \(\kappa\) (chance-corrected), and pairwise consistency across augmented question pairs.

Key Contributions

  • A seven-category, progressively structured diagnostic benchmark targeting rotation, translation, viewpoint change, and multi-object perspective taking
  • 2,739 samples across 4 visual domains with four orthogonal controlled variation axes (reference frame, premise, symmetry, syntax)
  • Evaluation of 43 VLMs (7 proprietary including GPT-5, Gemini 2.5 Pro, Claude 4 Sonnet; 36 open-source from 1B–38B)
  • Identification of systematic failure modes: egocentric bias, rotational blindness, and inconsistency under logically equivalent reformulations
  • Empirical scaling analysis showing both smooth improvements and emergent capabilities; weak correlation with prior spatial benchmarks confirming diagnostic novelty

Results

  • Human accuracy: 91.2%; human response time strongly correlates with VLM accuracy across tasks
  • Top proprietary model: GPT-5 ranks 1st in overall accuracy and pairwise consistency; Gemini 2.5 Pro ranks 2nd
  • Top open-source model: InternVL3-38B ranks 3rd overall and 2nd in consistency
  • Object-relation grounding: many models achieve \(\kappa > 0.6\); easiest category
  • Identity matching: bimodal—small models near chance, large models near perfect; suggests emergent scaling threshold
  • Mental rotation and perspective taking: most models at or below chance (\(\kappa \approx 0\)); hardest categories
  • Dynamic rotation: substantial difficulty across all model sizes
  • Consistency: severe inconsistencies on logically equivalent (symmetrical/syntactic) reformulations across nearly all models
  • SpinBench shows weak correlation with prior spatial benchmarks (e.g., OmniSpatial, MindCube), confirming it probes distinct spatial skills

Limitations

  • All viewpoint changes are restricted to horizontal orbits in a 2D plane; vertical spatial relations (above/below, height differences) are excluded, limiting generalization to full 3D reasoning
  • Benchmark is predominantly multiple-choice, potentially allowing partial credit from elimination strategies
  • Perspective taking tasks are limited to two subtypes (scene selection S and relation transformation T); more compositional variants are not covered
  • Evaluation of chain-of-thought (CoT) is restricted to three specialized spatial reasoning models; broader CoT analysis across standard VLMs is absent
  • Face domain images are restricted to 100 individuals Ă— 8 poses, limiting pose-space coverage

Relevance to Vision-Language Models

SpinBench directly probes whether VLMs possess the geometry-aware internal representations required for perspective taking—a capability prerequisite for embodied applications such as robotic manipulation and navigation. The finding that even frontier models (GPT-5, Gemini 2.5 Pro) perform near chance on mental rotation and perspective taking suggests that scale alone does not confer analog spatial simulation, a capability humans rely on. The premise-variation paradigm (visual vs. linguistic grounding conditions) provides a principled diagnostic for decomposing whether VLM spatial failures originate in visual perception or in geometric inference chains. For researchers building or fine-tuning VLMs for spatially grounded tasks, SpinBench offers a principled, cognitively motivated failure taxonomy that complements task-level spatial benchmarks.