Skip to content

SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPAZER is a training-free VLM-driven agent for zero-shot 3D visual grounding that fuses spatial (3D) and semantic (2D) reasoning through a coarse-to-fine pipeline: holistic multi-view scene rendering → retrieval-augmented candidate screening → 3D-2D joint decision-making. It eliminates the need for 3D-labeled training data while surpassing all prior zero-shot methods by 9.0% on ScanRefer and 10.9% on Nr3D.

Problem

Existing zero-shot 3DVG methods specialize in either spatial reasoning (3D-based: point clouds → textual descriptions fed to LLMs) or semantic reasoning (2D-based: video frames matched via VLMs), but not both. 3D-based methods fail on fine-grained attribute discrimination; 2D-based methods lack precise spatial awareness. This dichotomy limits performance on queries that jointly require spatial relations ("under the first table near the door") and visual attributes ("brown leather").

Method

SPAZER operates in three progressive stages, with GPT-4o as the VLM backbone:

  1. 3D Holistic View Selection: The input point cloud is rendered into n+1 global views (1 BEV + n uniformly distributed oblique views at 45° elevation). The VLM selects the single optimal viewpoint I*_3d that best observes the query-described object.

  2. Candidate Object Screening: A pre-trained 3D detector provides object bounding boxes and class labels. Text-based similarity matching filters candidates to the target class. When matching confidence falls below threshold Ï„, a visual retrieval-augmented fallback constructs a stitched "visual object table" of cropped object images and asks the VLM to reclassify. Surviving objects are annotated with IDs on the selected rendered view; the VLM outputs the Top-k (k=4) most likely candidates.

  3. 3D-2D Joint Decision-Making: For each of the k candidates, the most informative camera frame is retrieved by projecting the 3D bounding box's 9 keypoints onto all scan frames via P_2d = K · T_cw · P_3d and selecting the frame maximizing visible projected points. The VLM then jointly reasons over the global 3D rendering (for view-independent spatial relations) and the k retrieved 2D close-ups (for color, shape, material, view-dependent relations) to output the final bounding box.

Key Contributions

  • Training-free 3DVG agent integrating spatial and semantic reasoning without any 3D-labeled supervision.
  • Holistic multi-view 3D rendering that lets VLMs directly observe the scene instead of consuming textual coordinate descriptions.
  • Retrieval-augmented anchor filtering (RAF) with a visual object table fallback to handle ambiguous object category predictions.
  • Anchor-guided keyframe retrieval that avoids exhaustive video frame sampling, reducing inference time vs. prior 2D-based methods.

Results

ScanRefer (Acc@0.25 / Acc@0.5, zero-shot methods): - SPAZER: 57.2% / 48.8% overall - vs. VLM-Grounder (prior 2D SOTA): 51.6% / 32.8% → +5.6% / +16.0% - vs. CSVG (prior 3D SOTA): 49.6% / 39.8% → +7.6% / +9.0% - Matches or exceeds supervised baselines ScanRefer, 3DVG-Transformer, BUTD-DETR at Acc@0.25

Nr3D (selection accuracy, no GT class label): - SPAZER: 63.8% overall - vs. EaSe: 52.9% → +10.9% - vs. VLM-Grounder: 48.0% → +15.8% - With GT class label: 73.2% overall, surpassing CSVG (59.2%) and approaching Transcrib3D (70.2%)

Ablation highlights (Nr3D subset): - Removing JDM (3D only): 52.4% → adding JDM: 63.8% (+11.4%) - RAF vs. text-only filtering: +2–7% across sub-categories - HVS vs. random view: +3.6%; vs. BEV: +6.3% - Optimal k=4; optimal n=4 rendered views

Inference: 23.5 s/sample vs. VLM-Grounder's 50.3 s (2.1× faster)

Limitations

  • Grounding accuracy is bounded by the 3D object detector's localization and classification quality; errors in predicted bounding boxes propagate through the pipeline.
  • Camera-to-3D mapping is sensitive to inaccurate camera intrinsics/extrinsics and noisy depth maps.
  • Performance degrades as k grows beyond 4, suggesting the VLM is sensitive to the number of competing candidates in joint decision-making.
  • Relies on GPT-4o as default VLM; open-source alternatives (Qwen2-VL-72B) underperform by ~10 points overall, limiting fully open deployment.
  • Evaluated on ScanNet indoor scenes only; generalization to outdoor or synthetic 3D datasets is unverified.

Relevance to Vision-Language Models

SPAZER demonstrates a concrete paradigm for deploying VLMs as reasoning agents over 3D data without 3D-specific fine-tuning, directly relevant to research on zero-shot multimodal grounding and embodied reasoning. It shows that VLMs can handle multi-step spatial-semantic reasoning when given appropriate visual representations (rendered global views + close-up 2D crops) rather than text-only coordinate descriptions. The ablation confirming stronger VLMs yield proportionally better grounding underscores the importance of VLM capability for downstream spatial tasks, motivating continued investment in general VLM reasoning. The retrieval-augmented visual fallback pattern is broadly applicable to any VLM pipeline where symbolic labels are noisy.