Skip to content

HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

🕒 Published (v1): 2026-03-26 13:08 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; hierarchical 3D spatial understanding with perception-to-reasoning pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HiSpatial proposes a four-level hierarchical framework for 3D spatial understanding in VLMs, decomposing tasks from basic geometric perception to abstract spatial reasoning. An automated pipeline generates ~2B QA pairs from ~5M images, and a pointmap-augmented RGB-D VLM is fine-tuned on this corpus. The 3B-parameter model outperforms GPT-5 and Gemini-2.5-Pro on multiple spatial benchmarks.

Problem

VLMs excel at 2D vision-language tasks but lack systematic 3D spatial intelligence. Existing approaches to spatial SFT are fragmented: no unified task hierarchy captures the full range of spatial cognition, and large-scale 3D-grounded training data is scarce—indoor-only datasets lack coverage while in-the-wild web data lacks 3D supervision.

Method

The work defines four hierarchical cognitive levels: Level 0 (basic geometric perception: pixel-wise 3D point querying, pairwise depth ordering); Level 1 (object-level spatial understanding: localization, size, orientation); Level 2 (inter-object relational understanding: relative direction/distance, relational comparison); Level 3 (abstract spatial reasoning: perspective taking, spatial counting, spatial problem solving).

An automated pipeline processes ~5M images using MoGe-2 for metric-scale 3D point maps, RAM+GroundingDINO+SAM for object detection, OrientAnythingv2 for orientation, and Perspective Fields for gravity-aligned world coordinates. Textual references are generated by Describe Anything + Qwen2.5/3-VL and verified via IoU-matching re-grounding. GPT synthesizes Level-3 spatial problem-solving questions. The result is >2B QA pairs.

The model extends PaliGemma2-3B (SigLIP encoder + Gemma-2 LLM). A metric-scale 3D point map \(X \in \mathbb{R}^{H \times W \times 4}\) is encoded via sinusoidal positional encoding and a learnable Conv2D patchify layer, producing features that are concatenated with RGB image features along the channel dimension, projected linearly, and fed as fused visual tokens to the LLM. The visual encoder is frozen; the patchify layer, projector, and LLM are fine-tuned end-to-end via standard cross-entropy SFT.

Key Contributions

  • A principled four-level hierarchical taxonomy of 3D spatial intelligence for VLMs, with explicit inter-level dependency analysis.
  • An automated data pipeline generating >2B spatial VQA pairs from ~5M images (KosMos-2, Objects365, CA-1M) across all four levels.
  • A pointmap-augmented RGB-D VLM (HiSpatial-3B) achieving state-of-the-art results with only 3B parameters, surpassing GPT-5 and Gemini-2.5-Pro.
  • Empirical demonstration that lower-level task inclusion consistently improves higher-level reasoning performance (inter-level dependencies).

Results

  • SpatialRGPT-Quantitative (Level 1&2 avg.): HiSpatial-3B-RGB 72.43%, HiSpatial-3B (RGB-XYZ) 79.28% vs. GPT-5 40.47%, Gemini-2.5-Pro 26.57%, MM-Spatial-3B 68.70%.
  • QSpatial-Bench (Level 1&2 avg.): HiSpatial-3B 85.16% vs. GPT-5 68.45%, Gemini-2.5-Pro 49.92%, InternVL3.5-8B 57.94%.
  • CV-Bench-3D (Level 2): HiSpatial exceeds Gemini-2.5-Pro 90.80% (exact value not shown in provided excerpt for HiSpatial; surpassing claimed in abstract).
  • Inter-level ablation (Fig. 1): Removing Level 0&1 tasks drops Level 2 accuracy from 81.21 to 79.69; removing Level 1&2 drops Level 3 accuracy from 56.3 to 41.8.
  • HiSpatial-3B with GT point map further improves Level 1 width/height/distance tasks (e.g., vertical distance 90.16% vs. 86.89% estimated).

Limitations

  • Relies on off-the-shelf metric depth estimator (MoGe-2); errors in the estimated point map propagate into spatial reasoning.
  • Training data skews toward indoor scenes (CA-1M) and large-scale web images; coverage of outdoor or highly dynamic scenes may be uneven.
  • Visual encoder is frozen during fine-tuning, potentially limiting the model's ability to learn spatial-aware visual features from scratch.
  • Evaluation is primarily quantitative metric-accuracy–based; robustness to distribution shift or adversarial spatial queries is not assessed.
  • 3B parameter model; scaling behavior and applicability to larger LLM backbones are not explored.

Relevance to Vision-Language Models

HiSpatial directly addresses a core open challenge in VLM research: the gap between strong 2D language-vision alignment and weak 3D spatial grounding. The hierarchical SFT framework is directly applicable to any VLM backbone and provides principled guidance on task curriculum—the finding that lower-level tasks are prerequisite for higher-level reasoning is an actionable insight for future training recipe design. The pointmap-augmented architecture shows a clean, modality-agnostic way to inject metric 3D cues without full architectural redesign, relevant to any VLM practitioner working on embodied or robotics applications. The >2B-QA dataset and the explicit inter-level dependency analysis make this a strong foundation paper for the emerging subfield of spatially intelligent VLMs.