Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

🕒 Published (v1): 2026-05-28 17:00 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Yi Ma (Berkeley) + Joseph Tighe (Meta); CVPR 2026; 3D spatial priors improve VLM geometry

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GASP (Geometric-Aware Spatial Priors) injects 3D geometric inductive biases directly into VLM transformer layers via a lightweight correspondence head trained with dual supervision: InfoNCE contrastive loss on ground-truth point correspondences and a depth consistency regularizer. The head is discarded at inference, leaving a standard VLM that nonetheless learns view-invariant internal representations, yielding large gains on out-of-domain spatial benchmarks without any 3D VQA training data.

Problem

Standard VLMs have near-zero internal visual correspondence accuracy (<5% PCK), indicating the LLM backbone lacks geometric inductive bias. Prevailing remedies fail in complementary ways: fine-tuning on 3D VQA datasets induces dataset-specific bias memorization and poor OOD generalization, while integrating frozen specialized 3D encoders (e.g., VGGT) inflates model size and creates rigid feature-alignment bottlenecks. Neither paradigm teaches the model the underlying geometric consistency of the world.

Method

GASP attaches a lightweight 2-layer MLP correspondence head \(H_c\) to every transformer layer of the VLM's LLM backbone. Weights are initialized via SVD of the pre-trained query projection matrix of the same layer. During training, \(H_c\) maps visual tokens \(V^{(l)} \in \mathbb{R}^{N \times d}\) to correspondence embeddings \(E \in \mathbb{R}^{N \times d_{emb}}\) and is supervised by two complementary losses:

1. Contrastive correspondence loss \(L_{corr}\): InfoNCE over ground-truth point tracks from DL3DV, with all other patches in the target frame as negatives, teaching view-invariant 2D embeddings.
2. Depth consistency loss \(L_{depth}\): For each anchor point, the expected depth \(\hat{d}^b_i = \sum_j A_{ij} d^b_j\) is computed as a soft-argmax over the matching distribution \(A_{ij}\) from Eq. 6, then penalized against ground-truth depth using a scale-invariant \(\ell_1\) formulation: $\(L_{depth} = \frac{1}{N_{valid}} \sum_{i \in valid} \frac{|d^b_i - \hat{d}^b_i|}{d^b_i + \hat{d}^b_i + \epsilon}\)$ This acts as a discriminative geometric regularizer—penalizing visually similar but depth-dissimilar matches—rather than a depth estimator. The combined objective is: $\(L_{total} = L_{LM} + \lambda_c L_{corr} + \lambda_d L_{depth}\)$ \(H_c\) is fully discarded at inference; the model processes inputs as a standard VLM with no auxiliary 3D input. Training uses LoRA (rank 512) on Qwen2.5-VL-7B and LLaVA-NeXT-Video-7B for ~10h on 32 H200 GPUs, mixing ~1.75M generated geometric sequences with LLaVA-Video-178K to prevent catastrophic forgetting.

Key Contributions

• GASP framework: deep supervision of QK visual self-attention (\(Q_V K_V^T\)) across all LLM transformer layers via an auxiliary head that is removed at inference.
• Diagnostic analysis: reveals baseline VLMs have <5% PCK layer-wise and negative confidence-accuracy correlation (\(\rho \approx -0.22\)), diagnosing positional bias as a root cause.
• Dual geometric prior: combines 2D view-invariance (InfoNCE on point tracks) with 3D disambiguation (scale-invariant depth consistency via soft-argmax matching), applied jointly without 3D VQA data.
• Strong generalization: achieves large OOD gains while baselines fine-tuned on 3D VQA data degrade out-of-domain.

Results

• Layer-wise PCK: baseline <5% → GASP >70% (peak at middle-to-deep layers 20–25 for LLaVA, 25–28 for Qwen2.5-VL).
• Temporal robustness: baseline retains <5% of performance at 8-frame gap; GASP maintains >85% at 24-frame gap.
• Confidence-accuracy correlation: baseline \(\rho \approx -0.22\) (systematic miscalibration/positional bias) → GASP \(\rho \approx +0.62\) (well-calibrated).
• All-Angles Bench (camera pose estimation): +18.2% vs. controlled SFT baseline.
• VSI-Bench (object counting): +29.0% vs. controlled SFT baseline.
• BLINK (multi-view reasoning): +15.0% vs. controlled SFT baseline.
• Adding depth consistency (\(L_{depth}\)) consistently outperforms correspondence-only (\(L_{corr}\)) across all metrics, confirming its role as a geometric disambiguator.

Limitations

• Compute-intensive training: LoRA rank 512 across all 28–32 LLM layers on 32 H200 GPUs; not lightweight in absolute terms despite the head itself being small.
• Layer attachment and initialization heuristics (SVD of query projection, middle-to-deep layer peaking) may require re-tuning for architectures with different attention structures.
• Training data (DL3DV) consists of large-scale static scenes; performance on highly dynamic scenes or egocentric manipulation environments is not fully characterized.
• The correspondence head adds training complexity (additional losses, differential learning rates, negative sampling strategy) that is nontrivial to replicate across new VLM families.
• Results on general VQA benchmarks (Video-MME, TempCompass, NextQA) are mentioned but full numbers are not shown in the provided text excerpt, limiting assessment of catastrophic forgetting.

Relevance to Vision-Language Models

GASP directly targets the internal representation quality of VLM backbones rather than applying post-hoc spatial heads or VQA fine-tuning, showing that the \(Q_V K_V^T\) visual self-attention matrix is a meaningful diagnostic and optimization target for geometric understanding. The finding that baseline VLMs exhibit near-zero correspondence accuracy and negatively calibrated confidence is a strong diagnostic result that reframes the 3D spatial reasoning problem as an inductive-bias deficiency rather than a data-scale problem. The inference-transparent design—where the auxiliary head is discarded—makes GASP compatible with any downstream VLM pipeline without inference overhead or architecture changes. This approach generalizes the emerging paradigm of geometric self-supervision (previously explored in video diffusion models via QK-matching) to the VLM training regime, opening a pathway for richer geometric pre-training signals beyond web-scale text and image-caption data.