Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task¶
🕒 Published (v1): 2026-07-01 22:02 UTC · Source: Arxiv · Venue: ECCV 2026 · link
Why this paper was selected
Probes VLM vision vs. language bias on depth perception; diagnostic for building reliable VLMs
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
Current VLMs fail at basic depth ordering: all 12 tested models score between 47–56% accuracy—no better than random chance—on a controlled psychophysical benchmark isolating nine individual pictorial depth cues. The O3-D dataset (37K images, 147K question pairs) disentangles visual cue utilization from linguistic bias by varying both cue presence and referring-expression clarity, revealing that language bias dominates model responses far more than any depth signal.
Problem¶
Existing depth benchmarks neither isolate individual pictorial cues nor decouple vision from language contributions to VQA performance. Prior work (e.g., DepthCues) used real-world images where cues co-occur uncontrollably (60–80% of images share 4 dominant cues), preventing per-cue analysis. No existing metric quantifies whether a model's answer reflects the visual input or learned linguistic priors.
Method¶
The authors frame depth evaluation as an odd-one-out + depth-ordering psychophysical VQA task. Five same-class objects appear in each scene; one target sits on a different depth plane. Starting from a 3D scene rendered in Kubric, a base (ambiguous) 2D view is produced with no depth cues, then individual cues are injected one at a time by manipulating camera position, lighting, aperture, textures, or haze:
| Cue | Manipulation |
|---|---|
| Height-in-Plane (HP) | Camera translated vertically |
| Occlusion (OC) | Camera translated horizontally |
| Relative Size (RS) | Target moved along optical axis |
| Light-and-Shadow (LS) | Directional light added |
| Texture Gradient (TG) | Object textures added |
| Linear Perspective (LP) | Ground texture removed |
| Focusness (FO) | Camera aperture widened |
| Familiar Size (FS) | Real-world-sized objects substituted |
| Saturation (SA) | Haze equation applied |
This produces 9 single-cue and all \(\binom{9}{2}=28\) pairwise cue combinations. Question prompts vary along four axes: query template, target referring clarity (4 levels: saliency → depth → spatial → red-circle marker), distractor referring, and response instruction, yielding 1,026 unique prompt templates. The forced binary choice (closer/farther) avoids models defaulting to "unsure."
The novel SDGM metric (standard deviation of within-group means) quantifies vision-vs-language sensitivity: larger SDGM across cue-varied images signals visual sensitivity; larger SDGM across language-varied prompts signals linguistic sensitivity.
Near-far bias is formally defined as: $\(\mathrm{bias}_{NF}(M) = \mathrm{Acc}_F(M) - \mathrm{Acc}_N(M)\)$ where \(\mathrm{Acc}_F\) and \(\mathrm{Acc}_N\) are accuracy on target-far and target-near subsets respectively; zero indicates no bias.
Key Contributions¶
- O3-D dataset: 37,110 synthetic + real images and 147,552 image-question pairs with per-cue control (the only dataset isolating individual cues and their pairwise combinations).
- Synthetic pipeline: Kubric-based renderer supporting configurable objects (37 classes), environments (13), cue type, cue strength, and prompt variation.
- SDGM metric: Quantifies relative sensitivity of a VLM to vision vs. language inputs without requiring separate ablation models.
- Empirical finding: All 12 VLMs (open-source and commercial) operate at or below chance on depth ordering; language bias consistently outweighs visual cue utilization.
Results¶
- Overall accuracy: All 12 VLMs score 47–56%; DepthAnythingV2 baseline reaches 70–97% depending on image set. No VLM significantly exceeds chance (0.5).
- Best single cues for VLMs: Height-in-Plane (HP, mean 0.54) and Relative Size (RS, mean 0.54); gap vs. baseline: \(\delta_\text{HP}=0.45\), \(\delta_\text{RS}=0.37\).
- Real vs. synthetic: VLM mean on real mixed-cue images (0.61) vs. synthetic 1-cue (0.52) vs. synthetic 2-cue (0.53); DepthAnythingV2 scores 0.94/0.97/0.71 on the same splits.
- 0-cue (ambiguous) images: VLM mean 0.49 (below chance); DepthAnythingV2 0.52.
- ICL + CoT: Helps only commercial models—GPT-4.1-mini (+0.068), Gemini 2.5 Flash-Lite (+0.042); hurts DeepSeek (−0.022), Qwen2 (−0.012), Phi3.5 (−0.006). No model exceeds chance after prompting.
- Linear Perspective (LP): Removed LP (negative test) confirms positive interaction with other cues; average gain from LP presence ~0.03.
- Near-far bias: Wide spread across models; increasing cue strength does not consistently reduce bias in any VLM, unlike DepthAnythingV2 which shows monotone bias reduction.
Limitations¶
- Evaluation is limited to depth ordering (binary closer/farther); metric depth estimation or scene-scale reasoning is not tested.
- Synthetic scenes use a restricted set of 37 object classes and 13 environments from Kubric; real-world generalization of findings may vary.
- Real-world images cover only 3 object classes and 8 of 9 cues (Saturation excluded).
- SDGM as a sensitivity metric is descriptive and lacks a statistical test for significance of the vision-vs-language gap.
- The paper does not test VLMs with explicit 3D input modalities (e.g., point clouds, depth-channel prompting).
- Static image training data may be fundamentally insufficient for depth understanding; the paper identifies this but does not propose a remedy.
Relevance to Vision-Language Models¶
This paper provides direct, controlled evidence that VLMs—including state-of-the-art commercial models—do not meaningfully utilize pictorial depth cues in their visual reasoning, despite passing many coarser spatial benchmarks. The disentanglement methodology (psychophysical cue control + SDGM metric) is broadly applicable to diagnosing any visual capability claim in VLMs where language shortcuts could inflate apparent visual understanding. The finding that chain-of-thought and in-context learning fail to unlock depth reasoning challenges the common assumption that prompting strategies can compensate for weak perceptual grounding. For researchers studying visual grounding, spatial reasoning, and compositional scene understanding in VLMs, O3-D offers a rigorous diagnostic testbed that is significantly harder to game via language priors than existing benchmarks.