Skip to content

Grounded 3D-Aware Spatial Vision-Language Modeling

🕒 Published (v1): 2026-05-28 17:51 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Ligeng Zhu (MIT/NVIDIA); CVPR 2026; unified 2D+3D grounding in spatial VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GR3D is a spatial VLM that unifies explicit 2D grounding, implicit 2D grounding, and monocular 3D grounding in a single 8B-parameter framework. The key innovation is streaming region insertion: during chain-of-thought generation, the model dynamically predicts 2D bounding boxes for mentioned entities and immediately injects their visual embeddings back into the token stream before continuing reasoning. This grounding-as-inductive-bias approach achieves state-of-the-art on Omni3D 3D detection and multiple spatial VQA benchmarks.

Problem

Existing spatial VLMs lack two capabilities: (i) implicit 2D grounding — automatically linking entity mentions in free-form text to visual regions during generation without a separate detection phase — and (ii) robust monocular 3D grounding from a single view, which suffers from entangled depth/scale/intrinsics and the need to first resolve "which instance" before estimating 3D structure. Most systems either bypass the intermediate localization step, require multi-view supervision, or are constrained by sparse 3D box annotations.

Method

GR3D builds on NVILA-Lite-8B augmented with spatial positional encodings (2D pixel coordinates + relative depth) and a region encoder (feature pooling within a bounding box).

Explicit 2D grounding predicts bounding boxes as HTML-style text (<bbox>[x1,y1,x2,y2]</bbox>) via the standard language head — no separate detection branch.

Implicit 2D grounding via streaming region insertion: during autoregressive generation, whenever an entity is mentioned the model (a) predicts its 2D bbox coordinates as text tokens, (b) encodes the corresponding image crop into a region token via the region encoder, and (c) injects that region token directly into the ongoing token stream. Subsequent generation is conditioned on both language context and the freshly inserted visual evidence. Training uses teacher forcing with ground-truth regions; inference is fully autoregressive.

Monocular 3D grounding: grounded 2D region tokens serve as queries for 3D prediction. The 3D box is parameterized as \((x_c, y_c, z_c, w, h, l, \theta_p, \theta_r, \theta_y)\) in camera coordinates and predicted as text. Intrinsic-aware normalization rescales images by focal length (\(W' = \frac{1000}{f_x} W\)) to normalize field-of-view across datasets. Supervision combines sparse 3D-box labels, pure text-to-3D prediction for boxes without 2D annotations, and dense point supervision: ~100 random surface points per image (derived from depth maps) each paired with a region prompt, providing scalable geometric signal beyond scarce 3D annotations. Bounding box augmentation (jitter) adds robustness to localization noise.

Training is two-stage: Stage 1 spatial pretraining (visual encoder frozen, spatial PE + LLM trained on grounding + 3D detection data); Stage 2 CoT finetuning (LLM only, on Omni3D data reformatted into CoT format).

Data: 97K grounded CoT samples (built from RefSpatial + Florence-2 region proposal + VLM verification), 780K 3D detection samples (Omni3D + EmbodiedScan), 272K pointmap samples (DepthLM). No proprietary data.

Key Contributions

  • Streaming region insertion: a unified, single-stream mechanism that performs implicit 2D entity grounding on-the-fly during language generation, coupling perception and reasoning without discrete stage transitions.
  • Region-prompted monocular 3D grounding: decomposes 3D detection into (1) implicit 2D localization then (2) region-conditioned 3D regression, resolving both linguistic and geometric ambiguity.
  • Intrinsic-aware normalization: focal-length-based image rescaling to achieve consistent field-of-view representation across heterogeneous 3D datasets.
  • Dense point supervision: scalable depth-driven training signal that densifies 3D supervision far beyond sparse box annotations.
  • Demonstration that grounding is an inductive bias: grounding capabilities improve general spatial understanding even on tasks that do not require explicit localization.

Results

Omni3D 3D detection (Table 1): - GR3D-8B AP3D: 25.40 vs. DetAny3D 24.92 (best specialist), Qwen3-VL-8B 11.52 - GR3D-8B mAP: 31.64 vs. Qwen3-VL-8B 17.77 - Notably stronger on indoor datasets (SUN-RGBD AP15: 43.49 vs. 28.28 for Qwen3-VL-8B)

Omni3D 2D detection (Table 2): - GR3D-8B surpasses Cube R-CNN region proposals and Qwen3-VL-8B projected 2D boxes across all domains (e.g., SUN-RGBD mAP 38.86 vs. 15.07 for Cube R-CNN, 8.06 for Qwen3-VL-8B)

Spatial VQA (Table 3): - BLINK-Depth: 94.4% vs. SR-3D-8B 90.3%, SpatialRGPT-8B 87.9%, human 98.3% - CVBench (Rel./Dep./Dis. avg.): Stage 1 model 87.23 vs. NVILA-Lite-8B 86.31 - ERQA: Stage 1 model 76.00 vs. NVILA-Lite-8B 62.60 - SAT: Stage 1 model 81.01 vs. NVILA-Lite-8B 68.90 - General VQA benchmarks (ChartQA, POPE, AI2D) largely preserved after Stage 2 CoT finetuning

Limitations

  • Paper text is truncated before full ablation results and MMGCoT (grounding consistency) numbers are provided.
  • Stage 2 CoT finetuning shows slight regression on some spatial VQA benchmarks (e.g., ERQA drops from 76.00 to 70.60) relative to Stage 1.
  • Dense point supervision relies on depth estimation quality; errors in predicted depth maps propagate to 3D supervision.
  • Implicit grounding annotation pipeline uses Florence-2 + VLM verification, introducing noise that must be filtered — scaling to much larger corpora may amplify errors.
  • Single-view monocular 3D grounding is inherently limited for occluded or scale-ambiguous scenes; multi-view results are deferred to supplementary material and not fully reported.

Relevance to Vision-Language Models

GR3D directly advances the frontier of spatially-grounded VLMs by demonstrating that embedding explicit localization (both 2D and 3D) into the autoregressive generation process — rather than treating it as a post-hoc or separate module — yields stronger spatial reasoning across the board. The streaming region insertion mechanism is a practically important architectural pattern: it shows how region tokens can be woven into chain-of-thought without breaking the standard LLM generation interface, which is broadly applicable to any VLM that needs to ground its reasoning in visual evidence. For researchers tracking VLMs, the key insight that grounding acts as an inductive bias (improving tasks that don't explicitly require localization) has implications for how future spatial, embodied, and robotic VLMs should be trained. The intrinsic-aware normalization strategy also addresses a concrete calibration failure mode in current large VLMs (e.g., Qwen3-VL's resolution sensitivity) that is under-appreciated in the community.