Skip to content

When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning

🕒 Published (v1): 2026-07-13 07:13 UTC · Source: Arxiv · link

Why this paper was selected

Depth-ordinal prompting for VLM spatial reasoning; addresses a core 3D perception weakness in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Showing a depth map to a frozen VLM often degrades spatial reasoning because pseudo-depth introduces fine-grained ordering errors and the rendered image is hard for the model to down-weight. Depth-Ordinal Prompting (DOP) converts monocular depth into a single natural-language ordinal cue ("the chair appears closer than the table") appended to the text prompt, bypassing the visual stream entirely. This training-free interface recovers or exceeds depth-image performance across six VLM backbones and three benchmarks.

Problem

Frozen VLMs struggle with relative depth and spatial ordering despite encoding a usable depth signal in early visual layers. The straightforward fix—appending a rendered pseudo-depth heatmap as a second image—frequently makes performance worse (up to −27.4 pp on BLINK Relative Depth), not because depth is absent but because: (1) depth becomes linearly inaccessible in later LM layers (linear probe MAE grows 12.3× from L1 to L35 in Qwen3-VL), (2) monocular pseudo-depth makes structured, non-zero-mean ordering errors at fine-grained object level, and (3) the rendered map is a second visual input the frozen model cannot selectively ignore.

Method

Depth-Ordinal Prompting (DOP) is a training-free, label-free, model-agnostic pipeline:

  1. Run an off-the-shelf monocular depth estimator \(g\) on the RGB image \(I\) to obtain dense map \(D = g(I)\).
  2. For each queried object \(o_i\) with bounding region \(B_i\), compute an object-level depth readout: $\(d_i = \text{Pool}\{D(u) : u \in B_i\}\)$ (mean pooling; convention normalized so larger \(d_i\) = closer).
  3. Apply a question-conditioned ordinal operator \(\pi_q\) to map \((d_1,\ldots,d_k)\) into a binary relation \(r_q\) (e.g., \(o_i\) closer/farther than \(o_j\)).
  4. Verbalize \(r_q\) via \(\tau\) into a short text cue \(c_q\) (e.g., "the chair appears farther than the table").
  5. Predict via a single VLM inference: \(\hat{y}_\text{DOP} = \arg\max_y p_\theta(y \mid I,\, q \oplus c_q)\).

The depth map is never fed to the visual stream. Bounding boxes come from benchmark annotations (no detector required). The method requires no fine-tuning, adapter, feature injection, or auxiliary module.

The design is motivated by an ESDI (embedding-space depth injection) diagnostic: writing clean, linearly-structured CLIP depth features directly onto the LM's first-layer visual token positions improves indoor depth ordering by +13.1 pp over the RGB-only baseline on NYU, confirming the bottleneck is interface accessibility, not depth absence.

Key Contributions

  • Layer-wise probing showing depth is not erased but becomes nonlinearly entangled in later LM layers (12.3× degradation in linear MAE); causal ESDI intervention isolates this as the binding constraint.
  • Granularity analysis explaining when pseudo-depth images hurt: zero-mean region-level noise averages away (robust), but systematic object-level ordering errors are fatal at fine-grained tasks.
  • DOP: a minimal ordinal-text interface that exposes only the decision-relevant depth relation, with no added pixels, no 3D reconstruction, no global captions.
  • Empirical evidence of VLM self-regulation: the frozen model discounts the text cue when RGB evidence is confident, making explicit confidence gating unnecessary and always-on use effective.

Results

  • EmbSpatial-Bench: Qwen3-VL-8B 69.3 → 73.9 (+4.6 pp); gains of +24.3 pp (Qwen2-VL), +22.0 pp (LLaVA-1.6-Mistral), up to +26.7 pp (LLaVA-1.5-7B).
  • NYU region-pair task: Qwen3-VL-8B 75.5 → 97.2 (+21.7 pp); up to +55.5 pp (Qwen2.5-VL); ESDI diagnostic reaches +13.1 pp with GT depth features.
  • CV-Bench D+D (strong RGB regime): +0.9 pp (largely neutral), while depth-as-image gives −0.7 pp and raw depth values give −8.8 pp—confirming the interface diagnosis.
  • vs. training-free baselines on EmbSpatial: DOP (73.9%) significantly beats depth-as-image (+4.5 pp, \(p<10^{-3}\)), depth-as-text-values (+20.1 pp), CoT, bbox-in-prompt, and reference-object prompting; ties ByDeWay-LDP (73.0%, \(p=0.15\), n.s.) with a simpler cue.
  • Estimator generalization: DPT-Large gives +3.9 pp on EmbSpatial, −0.3 pp on CV-Bench—trend holds.
  • Gate ablations: oracle gate reaches 83.7% (upper bound); zero-shot and calibrated gates yield 72.6% and 72.1%, both below always-on DOP (73.9%).
  • Failure case: BLINK Relative Depth (abstract point markers, no object boxes): −22.6 pp, ordering accuracy ~58% (near chance).

Limitations

  • Requires queried-object bounding boxes; without them DOP degrades to a scene-level cue that is largely neutral (+0.6 pp on VSR, +0.4 pp on 3DSR-Bench).
  • On BLINK Relative Depth, pseudo-depth ordering accuracy near 58% forces a misleading cue (−22.6 pp); method is unreliable when depth estimators cannot resolve fine-grained point-level orderings.
  • ESDI (the causal diagnostic) presupposes GT-quality depth and feature-level access; it is not a deployable method.
  • Evaluated only on 7–8B frozen models; behavior under larger, instruction-tuned, or fine-tuned VLMs is untested.
  • Always-on ordinal cue is binary; multi-object (\(k>2\)) or continuous-ordering questions are not explicitly addressed.
  • No analysis of failure modes from bounding box quality when using a detector rather than benchmark annotations (only evaluated in an appendix stress test).

Relevance to Vision-Language Models

DOP directly addresses a core VLM capability gap—spatial and depth reasoning—without any model modification, making it immediately applicable to any frozen VLM. The central finding, form dependence, is a mechanistic insight with broad implications: the same auxiliary signal can be harmful or helpful depending solely on its modality interface (visual vs. textual), which challenges the common assumption that "more information = better" for VLMs. The layer-wise probing and ESDI causal intervention provide a reproducible framework for diagnosing where geometric information is lost in VLM computation, applicable beyond depth to other geometric or relational signals. For researchers building spatial, embodied, or multimodal reasoning systems on top of frozen VLMs, DOP establishes a lightweight, competitive baseline that is simpler than scene-captioning alternatives (ByDeWay) and avoids the cost of depth-supervised fine-tuning pipelines.