3D Visual Illusion Depth Estimation¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SOTA monocular and stereo depth estimators are systematically fooled by 3D visual illusions (flat surfaces rendered to appear 3D). The authors introduce a large-scale benchmark dataset (~3k scenes, 200k images, five illusion types) and a VLM-driven fusion framework that uses QwenVL2-7B commonsense reasoning to generate confidence maps that adaptively blend stereo disparity with monocular depth.

Problem¶

Monocular depth methods rely on texture cues (shape, shadow, perspective) that 3D illusions deliberately exploit, causing gross errors on inpainting, picture, replay, and holography illusions. Stereo methods depend on pixel-wise correspondence, which collapses on mirrors and transparent surfaces where reflections create ambiguous matches. No prior unified approach handles the full spectrum of illusion types, and no benchmark existed to expose these failure modes systematically.

Method¶

The framework has two components:

Dual-branch prediction network: A GRU-based stereo branch constructs a cost volume from rectified image pairs and iteratively refines disparity. In parallel, frozen DepthAnything V2 predicts affine-invariant monocular depth; its features also guide stereo refinement via learnable convolutions.

VLM-based fusion network (three stages): 1. VLM prediction: QwenVL2-7B receives the left RGB image, binocular disparity map, and monocular disparity map alongside a language prompt querying transparent/reflective material presence (with corner-coordinate outputs). The LLM decoder produces embedding features encoding scene-level reliability cues. 2. Confidence map generation: A flow-matching module (Flux-inspired) iteratively transforms Gaussian noise to a structured confidence distribution conditioned on concatenated image+text embeddings via Transformer cross-attention; the final state is decoded by a VAE and combined with a cost volume via convolution to output a per-pixel confidence map. 3. Global fusion: Affine parameters (scale, shift) align affine-invariant monocular depth to metric scale, pooling from high-confidence neighbors to suppress unreliable regions. The aligned monocular depth, stereo disparity, and confidence map are fused via convolution+upsampling to produce the final depth map.

Key Contributions¶

3D-Visual-Illusion dataset: ~3,000 scenes, >200,000 frames covering inpainting, picture, replay, holography, and mirror illusions; combines web-sourced video (with RANSAC plane-fitting depth correction and SAM2 segmentation), generative model outputs (Sora, Kling, HunyuanVideo reconstructed via InstantSplat/DUSt3R+GS), and real-world stereo+LiDAR capture (ZED Mini + Realsense L515).
Systematic failure-mode analysis of monocular (DepthAnything V2, Marigold, DepthPro, Metric3D), multi-view (DUSt3R, VGGT), and stereo (RAFT-Stereo, Selective-IGEV, MochaStereo) methods across illusion types, documenting complementary failure patterns.
VLM-driven confidence map via flow matching that converts VLM embeddings into pixel-space reliability estimates for depth fusion.
Demonstration that standard fine-tuning on illusion data fails for both monocular and stereo models due to conflicting priors, motivating the VLM fusion approach.

Results¶

Real-world 3D-Visual-Illusion dataset (Table 1): Ours achieves AbsRel 0.03 and δ1 99.60, matching the best stereo baseline (Selective-RAFT) while substantially outperforming all monocular methods (best monocular + GT-alignment: AbsRel 0.09).
Booster dataset, transparent regions (Table 2, zero-shot): Ours EPE 7.32, bad2 56.77 vs. best prior stereo (MochaStereo: EPE 9.18, bad2 66.64); also best on non-transparent regions (EPE 1.76, bad2 10.06 vs. MochaStereo 2.82/12.25).
Middlebury dataset (Table 5, standard stereo benchmark): Ours EPE 1.50 vs. best baseline RAFT-Stereo 1.92, showing no regression on non-illusion scenes.
Ablation (Table 6, Booster transparent): Stereo-only baseline EPE 15.11/bad2 80.38; adding VLM fusion reduces to EPE 7.32/bad2 56.77 — bad2 improves by >10 points over the next-best fusion variant.
VLM confidence map error rate ~20% in zero-shot generalization on Booster.

Limitations¶

Virtual data generation requires manual semantic segmentation with SAM2 (labor-intensive; automatic illusion detection fails because VLMs themselves can be fooled).
Real-world subset covers only three illusion types (inpainting, picture, replay); holography and mirror are absent from real captures.
VLM-driven fusion (QwenVL2-7B) is computationally expensive at inference.
Evaluation restricted to pure, non-composite illusions; combinations of multiple illusion types or novel illusion categories are not addressed.

Relevance to Vision-Language Models¶

This paper offers a concrete case study of deploying a large VLM (QwenVL2-7B) as a geometric reasoning module rather than for language or QA tasks — the VLM's commonsense knowledge about material properties (mirrors, glass, transparency) is distilled into per-pixel confidence maps via flow matching, bridging high-level semantic understanding and low-level depth geometry. It demonstrates that VLM features encode scene-level cues (reflectiveness, transparency) that purely discriminative stereo or monocular models cannot capture, contributing to the growing literature on VLMs as universal perception priors. The flow-matching bridge from VLM embedding space to dense spatial confidence maps is a transferable architectural pattern for other pixel-level tasks where high-level reasoning must guide low-level prediction.