Skip to content

Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Egocentric multi-view spatial reasoning benchmark; VLMs struggle with 3D

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Ego3D-Bench is the first ego-centric multi-view outdoor benchmark (8,600+ QA pairs) for evaluating VLM 3D spatial reasoning, drawn from nuScenes, Waymo, and Argoverse. Benchmarking 16 SOTA VLMs reveals a large human-VLM performance gap. Ego3D-VLM, a training-free plug-and-play framework that augments any VLM with a textual cognitive map derived from metric depth and referring-expression comprehension, yields 12% and 56% average relative improvements on multi-choice accuracy and absolute distance RMSE, respectively.

Problem

Prior spatial benchmarks target single images or indoor monocular video, neither of which matches the ego-centric multi-view observations (simultaneous front/side/rear cameras) used by real embodied agents such as self-driving cars and mobile robots. Existing work does not evaluate VLMs on structured, directional, temporally evolving multi-view inputs in dynamic outdoor scenes, leaving a critical blind spot for embodied-AI spatial understanding.

Method

Ego3D-Bench construction. QA pairs are sourced from nuScenes (6 cameras), Waymo (7 cameras), and Argoverse 1 (5 cameras). Human annotators filter unique objects per scene, write discriminative captions, and collect ground-truth 3D bounding boxes. Rule-based templates generate answers; motion-reasoning pairs receive an additional human review. Five task categories: (1) absolute distance measurement (multi-choice and metric RMSE), (2) relative distance, (3) localization, (4) motion reasoning, (5) travel time.

Ego3D-VLM pipeline. Given multi-view images \(\{I^{(v)}\}\) and a query \(q\): 1. REC: Grounding-DINO-Base localizes each referred expression as 2D bounding boxes \(O^{(v)} = \{(b_i^{(v)}, c_i^{(v)})\}\). 2. Metric depth: DepthAnything-V2-Metric-Large produces dense \(D^{(v)} \in \mathbb{R}^{H \times W}\); depth \(d_i^{(v)}\) is sampled at each box center. 3. 3D lift: Each center is projected to camera coordinates via \(p_{\text{cam},i}^{(v)} = d_i^{(v)} \cdot (K^{(v)})^{-1} [x_i, y_i, 1]^\top\). 4. Global transform: All views are transformed to the front-camera coordinate frame using \((R^{(v)}, T^{(v)})\). 5. Relational scaling: 3D points are scaled by \(s = h_{\text{cs}} / h_{\text{est}}\) using known canonical heights of familiar categories (e.g., 1.7 m for humans) to produce physically plausible metric estimates without ground-truth depth. 6. Cognitive map: \(\mathcal{C} = F_{\text{cog}}(\{p_{\text{scaled},i}^{(v)}, c_i^{(v)}\}_{i,v})\) formats object positions and originating viewpoints as a structured text, which is prepended to the VLM input alongside the images: \(a = \mathcal{V}(\mathcal{C}, \{I^{(v)}\}, q)\).

Key Contributions

  • Ego3D-Bench: 8,600+ QA pairs across 5 task categories (ego- and object-centric) from three prominent outdoor multi-view driving datasets; first ego-centric multi-view benchmark for VLMs.
  • Ego3D-VLM: Training-free, model-agnostic framework that constructs a textual cognitive map from off-the-shelf REC and metric depth models—no fine-tuning, no point-cloud reconstruction.
  • Comprehensive evaluation of 16 SOTA VLMs (GPT-4o, Gemini-1.5-Pro, InternVL3, Qwen2.5-VL, Ovis2, SpatialRGPT, SpaceThinker) establishing human and chance baselines.
  • Demonstrated compatibility with specialized 3D-VLMs (SpaceThinker-Qwen2.5-3B) in addition to generalist models.

Results

  • Human vs. VLM gap: Human average accuracy 85.3%; best unaided VLM (Qwen2.5-72B) achieves 58.0%; best Ego3D-VLM-augmented result (Qwen2.5-72B) reaches 69.5%—still ~16 points below human.
  • Ego3D-VLM gains (generalist VLMs): Average 12% relative improvement in multi-choice accuracy and 56% relative reduction in absolute distance RMSE across all tested models.
  • GPT-4o: 56.7% → 73.2% accuracy; RMSE 19.2 → 7.4 m
  • Gemini-1.5-Pro: 57.5% → 73.1%; RMSE 19.6 → 7.2 m
  • InternVL3-78B: 59.9% → 71.8%; RMSE 13.8 → 7.4 m
  • Qwen2.5-3B (smallest): 39.1% → 44.4%; RMSE 32.1 → 12.6 m
  • 3D-VLM baseline: SpaceThinker-Qwen2.5-3B achieves 46.2% avg, outperforming SpatialRGPT-8B (44.0%); Ego3D-VLM adds +3% accuracy and reduces RMSE by >4 m.
  • Blind VLM: Text-only GPT-4o/Gemini average 53.8%—only 5% below vision-enabled, confirming spatial reasoning largely from world priors rather than visual understanding.
  • VLM vs. LLM with cognitive map: Full VLM (60.1%, RMSE 8.0) outperforms LLM-only backbone (53.1%, RMSE 11.4) with the same cognitive map, showing VLMs exploit image cues to tolerate REC/depth errors.

Limitations

  • Performance still substantially below human on localization (40–45% for best VLMs even with cognitive map) and travel time tasks; cognitive map does not fully resolve multi-view integration failures.
  • Cognitive map quality is upper-bounded by REC and metric depth accuracy; errors in object localization or depth estimation propagate to downstream spatial reasoning.
  • Relational scaling assumes detection of familiar categories (sedans, humans, bikes) per scene; failure to detect these degrades metric estimates.
  • Benchmark is restricted to outdoor driving datasets; generalizability to indoor or aerial ego-centric multi-view setups is not evaluated.
  • Inference overhead introduced by external REC and depth models is not fully characterized relative to direct VLM inference in the main paper (only referenced as significantly smaller than point-cloud methods).

Relevance to Vision-Language Models

This paper directly targets a critical gap in VLM evaluation: the disconnect between passive single-view/indoor benchmarks and the structured multi-view perceptual setting of embodied AI. It provides concrete evidence that even the largest current VLMs (72B+) cannot construct coherent ego-centric world models unaided, pointing to a fundamental architecture limitation rather than a scale problem. Ego3D-VLM's training-free cognitive map approach offers a practical, model-agnostic probe and plug-in for any VLM being deployed in robotics or autonomous driving contexts. The results also establish nuanced findings relevant to VLM spatial research—object-centric absolute distances are tractable with structured 3D context (VLMs can surpass humans), while localization from a third-party perspective remains a persistent hard case.