Skip to content

SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SD-VLM addresses the inability of VLMs to perform quantitative 3D spatial reasoning (e.g., measuring object dimensions or inter-object distances) by combining a large-scale metrically accurate spatial QA dataset with a lightweight depth positional encoding (DPE) module. The model achieves 56.31% on the authors' new MSMU-Bench, outperforming GPT-4o by 26.91% and InternVL3-78B by 25.56%.

Problem

Existing VLMs excel at 2D semantic tasks but fail at quantitative 3D spatial queries (absolute distances, physical dimensions) because: (1) training data lacks precise metric spatial annotations—prior datasets rely on estimation models that introduce systematic errors rather than ground-truth 3D annotations; and (2) 2D image inputs lack the depth cues needed to reconstruct 3D structure without camera intrinsics.

Method

Two components are introduced:

MSMU Dataset: Generated from ScanNet/ScanNet++ 3D point clouds. A scene graph captures per-object 3D bounding boxes with real-world metric coordinates; instances are rasterized onto 2D images to transfer accurate annotations. Template-based QA generation yields 700K pairs across 8 task categories (scale estimation, grounding, distance, relative position, scale comparison, reference-object estimation, counting, existence). 10K CoT pairs are produced by prompting Qwen2.5-VL with a known reference-object measurement and filtering reasoning chains with DeepSeek-V3 for factual/logical consistency.

Depth Positional Encoding (DPE): The depth map D ∈ ℝ^{H×W×1} is patch-pooled to match the CLIP ViT patch grid, producing D′. Sinusoidal positional embeddings (same form as transformer positional encoding but along the z-axis, frequency base 10000) are computed per patch and added to the CLIP image feature map: E_vision = E_image + E_depth. This requires no extra tokens, no additional connector modules, and no change to the LLM. At inference without ground-truth depth, Depth-Anything-V2 supplies the depth map. The full model fine-tunes LLaVA-1.5-7B with LoRA on MSMU for one epoch (32 V100 GPU-hours).

Key Contributions

  • MSMU dataset: 2K scenes, 25K images, 700K QA pairs, 2.5M metrically accurate numerical annotations; the only spatial dataset covering all eight task categories including reference-object estimation.
  • MSMU-Bench: Held-out benchmark of ~1K QA pairs from unseen scans, scored by GPT-4 with a δ = max(d̂/d, d/d̂) ≤ 1.25 success-rate threshold.
  • Depth Positional Encoding (DPE): Adds 3D spatial priors via element-wise addition to image features—no extra tokens, no architectural modules beyond pooling + sinusoidal encoding.
  • Formal proof (Section A) that ≥4 known 3D distances in an image suffice to implicitly calibrate camera intrinsics, motivating the dataset's metric annotation strategy.

Results

  • MSMU-Bench: SD-VLM achieves 56.31% average success rate vs. GPT-4o 29.17% (+26.91%), InternVL3-78B 30.75% (+25.56%), SpatialRGPT 28.98%, SpatialBot 29.17%; adding MSMU-CoT raises to 59.19%.
  • Reference-object estimation: 40.00% vs. next-best 20.00% (Gemini-2).
  • Existence task: 87.23% (SD-VLM) vs. 44.68% (GPT-4o).
  • Q-Spatial++: 56.2% vs. InternVL3-78B 53.6%, GPT-4o 52.0%, SpatialRGPT 43.5%.
  • SpatialRGPT-Bench quantitative: 33.3% vs. SpatialRGPT 28.7%, GPT-4o 13.0%.
  • SpatialRGPT-Bench qualitative: 65.5% vs. InternVL3-78B 62.2%, SpatialRGPT 57.8%.
  • Depth integration ablation (MSMU-Bench): depth-as-image 22.64%, depth-as-prompt 48.78%, depth-as-token 35.72%, DPE (sincos) 56.31%; baseline image-only 46.73%.
  • General benchmarks: SD-VLM matches or slightly improves over LLaVA-1.5-7B on Whatsup, GQA, VQA-v2 while MSMU-Bench improves from 19.5% to 55.8%.
  • DPE noise robustness: At Gaussian noise δ=0.7, performance is 51.4% (vs. 56.3% no noise, 46.7% no depth).

Limitations

  • Dataset restricted to indoor 3D scans (ScanNet/ScanNet++); generalization to outdoor scenes or arbitrary RGB-D sources is unverified in the training pipeline (though tested at inference on SpatialRGPT-Bench outdoor scenes).
  • Relies on an external metric depth estimator at inference; depth quality directly affects performance (estimated depth trails GT depth; UniDepth underperforms DepthAnything-V2 by ~1 point average).
  • Base model is LLaVA-1.5-7B; no experiments with larger or more recent VLM backbones.
  • MSMU-Bench is small (~1K samples); GPT-4-based scoring introduces evaluator variance.
  • Metric spatial annotations require 3D scene data, limiting dataset scalability beyond existing indoor scan repositories.

Relevance to Vision-Language Models

SD-VLM directly targets a known blind spot of VLMs—quantitative 3D spatial reasoning—and demonstrates that it can be addressed without architectural overhaul by injecting depth as positional bias rather than as extra tokens or modalities. The DPE technique is minimally invasive and composable with any ViT-based VLM, making it practically relevant to anyone fine-tuning or augmenting current VLM architectures. The MSMU dataset and benchmark fill a gap where prior spatial datasets either lacked metric accuracy (model-pipeline-generated labels) or task diversity (no reference-object or existence tasks). For researchers tracking VLMs, this establishes a clear methodology: metric 3D supervision + lightweight spatial encoding yields large gains on quantitative spatial tasks with no regression on general VQA.