Skip to content

SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning

🕒 Published (v1): 2026-03-28 22:49 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; layered geometry-language fusion for reliable 3D spatial VLM reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatialStack proposes hierarchical geometry–language fusion for 3D VLMs, injecting multi-level features from a geometry encoder (VGGT) as additive residuals into successive LLM decoder layers rather than fusing only into the vision pathway. A systematic layer-wise analysis reveals that shallow geometry features support fine-grained perception while deep features support high-level relational reasoning, and that naive multi-layer fusion into the vision encoder causes feature interference. VLM-SpatialStack (built on Qwen3.5) achieves state-of-the-art performance on VSI-Bench and multiple other 3D spatial benchmarks.

Problem

Existing VLMs that integrate multi-view geometry encoders (DUST3R, VGGT) fuse only the final deep-layer features of those encoders with vision tokens. This single-level fusion discards hierarchical geometric signals: shallow encoder layers encode fine-grained local structure and boundaries, while deep layers become spatially homogeneous, losing the precise cues needed for low-level tasks (depth comparison, distance estimation). No prior work systematically studies which fusion depth is appropriate for which task granularity, nor how to combine levels without interference.

Method

SpatialStack extracts intermediate patch-token features \(Z_{l_i}\) from VGGT layers \(\{l_1, l_2, l_3\} = \{11, 17, 23\}\) (counted from zero). Each is passed through a layer-specific geometry token merger \(M_\text{geo}^{(l_i)}\) that aligns spatial resolution and embedding dimension to the LLM hidden size:

\[G_{l_i} = M_\text{geo}^{(l_i)}(Z_{l_i}), \quad G_{l_i} \in \mathbb{R}^{N' \times D_\text{lang}}\]

The three geometry features are injected as additive residuals into LLM decoder layers \(\{0, 1, 2\}\):

\[H^{(j)} \leftarrow H^{(j)} + G_{l_j}, \quad j \in \{0, 1, 2\}\]

Crucially, fusion occurs on the language side (geometry–language fusion, GlF) rather than into the vision pathway (geometry–vision fusion, GVF), which was found to cause feature interference. The vision and geometry encoders are frozen; only the fusion projectors and LLM decoder are updated with a standard next-token cross-entropy objective. VLM-SpatialStack is instantiated on Qwen3.5 (primary) and Qwen2.5-VL (for fair comparison with prior baselines).

Key Contributions

  • First systematic layer-wise ablation showing that geometry–language fusion follows a hierarchical correspondence: layer 11 peaks for low-level tasks (fine-grained perception), layer 23 peaks for high-level tasks (relational reasoning), and naive multi-layer vision-pathway fusion causes interference rather than synergy.
  • SpatialStack: a general, model-agnostic framework for progressively stacking multi-level geometry features into LLM decoder layers.
  • VLM-SpatialStack achieving SOTA on VSI-Bench, SPAR-Bench, and CV-Bench among open-source models, with strong cross-task generalization.

Results

  • VSI-Bench (high-level spatial reasoning, Table 3): SpatialStack-5B scores 67.5 avg (rank 1 open-source), vs. SpatialStack-4B (Qwen2.5) 60.9, Cambrian-S-3B 57.3, Qwen3.5-4B fine-tuned 53.6, VG-LLM-4B 47.3, Spatial-MLLM-4B 47.0, GPT-4o 34.0, Gemini-2.5 Pro 51.5.
  • Cross-benchmark ablation (Table 2, overall avg): SpatialStack 65.35 vs. GVF-L23 (VG-LLM equivalent) 65.30 vs. GVF-L11/17/23 64.83 vs. Qwen3.5 fine-tuned 63.07; highest on VSI-Bench, SPAR-Bench, and CV-Bench columns.
  • SpatialStack achieves the highest High-Level Avg (66.36) across ablation conditions, outperforming single-layer alternatives.
  • On BLINK-Spatial, the base Qwen3.5 fine-tuned model remains strongest; geometry fusion variants including SpatialStack do not exceed it there.

Limitations

  • Geometry-language fusion does not universally dominate: BLINK-Spatial performance regresses relative to the fine-tuned base model, indicating benchmark-dependent sensitivity.
  • Ablation shows naive multi-layer GVF hurts performance; the correct injection site (LLM decoder) requires careful architecture choice.
  • Requires multi-view image input compatible with VGGT; single-image spatial reasoning is not directly addressed.
  • Computational overhead of three additional layer-specific projectors and the VGGT encoder is not analyzed.
  • Evaluated only on Qwen-family base models; generalizability to other LLM backbones is asserted but not empirically demonstrated.

Relevance to Vision-Language Models

SpatialStack directly targets the well-documented failure of VLMs at 3D spatial reasoning by revealing a principled connection between geometry encoder depth and task granularity—a finding analogous to the established hierarchy in vision encoders. The key architectural insight, injecting into LLM decoder layers rather than the vision pathway, offers a concrete design rule for future 3D VLM work. This extends the DeepStack token-stacking paradigm to the geometry modality and provides a reusable plug-in framework compatible with any open-source VLM. For researchers tracking VLMs, it establishes a new SOTA on VSI-Bench and frames multi-level geometry–language fusion as a necessary ingredient for embodied and physical AI applications.