Skip to content

SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SSR introduces a plug-and-play module (MIDI) that converts monocular depth maps into structured textual rationales, then distills those rationales into compact latent tokens injected into existing VLMs to improve spatial reasoning. The authors also release SSR-CoT, a 1M+ sample depth-aware CoT dataset, and SSRBench, a multi-task spatial reasoning benchmark. Experiments show consistent gains over Qwen2.5-VL backbones on both spatial and general VQA tasks.

Problem

VLMs trained on RGB inputs lack precise spatial understanding (relative distances, object positions, depth relationships). Prior approaches either require specialized sensors (LiDAR, point clouds) or treat depth as a raw additional input without leveraging its inferential value for higher-order reasoning — depth is appended but not reasoned over.

Method

MIDI (Mamba-based Image-Depth Interpreter): Given an RGB image, Depth Pro produces a metric depth map. CLIP ViT-L/14 encodes the RGB image; SigLIP encodes the depth map. Both are projected via MLP into a shared semantic space. A Mamba-based language model fuses visual tokens, depth tokens, and the textual query to produce latent rationale tokens Z_R. A latent projector maps Z_R into the VLM's word embedding space.

Knowledge distillation: In Stage 1, a frozen LLM is trained to reconstruct full textual rationales from the MIDI-produced latent tokens (causal LM loss on rationale Y_R). This forces MIDI to compress spatial reasoning into the latent representation. In Stage 2 (optional), MIDI and the VLM are jointly fine-tuned with LoRA on the answer-only objective, discarding explicit rationale supervision.

Inference: Z_R latent tokens are prepended to the query sequence fed into the VLM — no architectural changes to the VLM required (plug-and-play).

SSR-CoT dataset: ~1.2M image-depth-question-rationale-answer tuples assembled from LLaVA-CoT, Visual-CoT, VoCoT, and SpatialQA. SpatialRGPT mines bounding boxes, object sizes, and distances; GPT-4o generates detailed spatial rationales. A quality filter retains only rationales that improve VQA accuracy on a held-out 10% cache sample.

SSRBench: Held-out benchmark with 6 tasks — Existence, Attribute, Action, Counting, Position, Object — split into General and Spatial categories. LLM-judge (Qwen2.5-14B-Instruct-1M) evaluates open-ended answers.

Key Contributions

  • SSR framework: depth-to-rationale conversion + knowledge distillation into plug-and-play latent tokens for existing VLMs
  • MIDI module: Mamba-based interpreter that jointly encodes RGB and depth conditioned on the query to generate spatial rationale embeddings
  • SSR-CoT: 1M+ scale visual-language CoT dataset with intermediate spatial reasoning annotations (bounding boxes, metric distances, object sizes)
  • SSRBench: comprehensive 6-task benchmark for spatial and general VQA evaluation, with dual LLM-judge validation (Qwen + GPT-4o-mini)
  • Demonstrated inference efficiency: latent reasoning reduces CoT token count from 437 to 2.62 tokens/second throughput improvement, cutting end-to-end inference time from 23.16s to 0.32s per sample

Results

  • SSR (3B) vs. Qwen2.5-VL-3B backbone on SSRBench Spatial: 69.7 vs. 48.8 (+20.9 average)
  • SSR (7B) vs. Qwen2.5-VL-7B on SSRBench Spatial: 76.1 vs. 53.6 (+22.5 average)
  • SSR (3B) on SpatialBench: 64.8 vs. 59.3 backbone (+5.4); SSR (7B): 67.0 vs. 64.7 (+2.3)
  • CV-Bench (out-of-domain): 68.9 vs. 67.0 (3B, +1.9); 73.3 vs. 73.0 (7B, +0.3)
  • VSR zero-shot: 82.9 vs. 76.4 (+6.5)
  • General VQA (3B): VQAv2 79.0 vs. 72.5 (+6.5); GQA 63.6 vs. 56.2 (+7.4); TextVQA 61.3 vs. 57.0 (+4.3)
  • SSR-CoT quality check: rationale-augmented answers improve Qwen2.5-VL-7B accuracy by 11.62% and score by 0.4568 on held-out subset
  • Plug-and-play (no Stage 2): average gains of 4.4 and 1.6 on SpatialBench and SSRBench respectively; full SSR adds further 5.7 and 18.7
  • SSR-3B matches or exceeds proprietary GPT-4o-mini and Claude-3.5-Haiku on several tasks

Limitations

  • Backbone restricted to the Qwen/Qwen2.5-VL model family; generalizability to other VLM architectures (e.g., LLaMA-Vision, InternVL) untested
  • Requires a monocular depth estimation preprocessing step (Depth Pro) at both training and inference time, adding a dependency on a separate model
  • Stage 2 training is compute-intensive (~48 hours on 8×H800 GPUs) and requires paired SSR-CoT data
  • SSRBench evaluation relies on LLM judges, which may carry subtle biases despite dual-model cross-validation
  • Improvement on out-of-domain benchmarks (CV-Bench) is marginal for the 7B variant (+0.3), suggesting saturation or limited transfer for certain tasks

Relevance to Vision-Language Models

SSR directly addresses the well-known weakness of VLMs in spatial/geometric reasoning by converting an underutilized modality (depth) into the token space VLMs natively operate in, rather than injecting raw depth features. The knowledge-distillation-into-latent-tokens approach generalizes the METEOR paradigm to a spatially-grounded setting and offers a modular upgrade path for deployed VLMs without full retraining. The work is relevant to any VLM application requiring scene understanding — robotics, embodied AI, autonomous driving — and the SSR-CoT dataset provides a large-scale resource for training depth-aware multimodal reasoning. For researchers tracking VLMs, the plug-and-play deployment strategy and the explicit spatial CoT framing represent a concrete design pattern for capability extension beyond text-image pretraining.