Skip to content

BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BeliefMapNav is a zero-shot object navigation system that replaces flat BEV value maps with a 3D voxel-based belief map, fusing hierarchical CLIP visual features and LLM-generated semantic landmarks to estimate target location probability across 3D space. A frontier observation belief module combines this prior with a visibility map to form a posterior, then a simulated-annealing planner minimizes expected search distance over frontiers. The system achieves state-of-the-art SR and SPL on HM3D and HSSD without any task-specific training.

Problem

Existing ZSON methods either use 2D BEV value maps that lack discriminative semantics and high-level reasoning, or query LLMs/VLMs directly for next-goal selection—both approaches fail to maintain a global 3D spatial posterior and suffer from greedy, back-and-forth navigation. VLMs are constrained by poor spatial reasoning and inability to extract fine-grained spatial information from images; LLMs lose precision when environment state is compressed into text descriptions.

Method

3D Voxel-Based Belief Mapping: A voxel grid (45,000 voxels at 0.25 m resolution) stores per-voxel belief scores. Each voxel accumulates CLIP visual features at three hierarchical semantic levels (scene, region, object) via multi-scale patch extraction scored by purpose-built hierarchical scorers (Scene/Region/Object Scorers weighting spatial volume, instance density, and point density respectively). GPT-4o generates nine hierarchical landmarks (3 per level: room, region, object) with per-landmark relevance scores α. Belief at voxel u is computed as the weighted sum of cosine similarities between CLIP text embeddings of landmarks/target and stored voxel features: bᵤ = Σ_lt Σ_i αᵢ^lt · pᵤ,ᵢ^lt + pᵤ,target.

Frontier Observation Belief Estimation: A visibility map p̂ᵥᵤ ∈ [0,1] encodes detection confidence per voxel based on distance and image boundary proximity—undetected high-confidence voxels are suppressed. The posterior b̂ᵤ^post = p̂ᵥᵤ · bᵤ is aggregated over each frontier's FOV via ray casting across four view directions; the max over directions gives the frontier's observation belief Pobs(xfi).

Observation Belief-Based Planning: A permutation π of frontier visits is optimized to minimize Σᵢ (cumulative A* path cost to frontier i) × Pobs(fπᵢ), solved via GPU-accelerated simulated annealing. The agent replans at every step using the updated posterior belief map.

Key Contributions

  • 3D voxel-based belief map integrating hierarchical CLIP visual features and GPT-4o-derived semantic landmarks with relevance scores for fine-grained prior estimation of target location.
  • Visibility-aware posterior belief update that suppresses revisitation of well-observed empty regions.
  • Distance-cost-aware belief-based planner over frontier sequences (via simulated annealing on A* path costs weighted by observation belief), replacing greedy next-goal selection.
  • SOTA results on HM3D and HSSD in the zero-shot setting without task-specific training.

Results

  • HM3D: SR 61.4% / SPL 30.6 — +3.4% SR and +9.7 SPL vs. InstructNav (best prior SR method); +6.9% SR and +5.5 SPL vs. UniGoal.
  • HSSD: SR 65.2% / SPL 32.1 — +14.2% SR and +7.2 SPL vs. best prior zero-shot method.
  • MP3D: SR 37.3% / SPL 17.6 — marginal +0.1 SPL improvement; SR slightly below UniGoal/SG-Nav due to MP3D mesh quality issues.
  • Ablation (HM3D, 400 episodes): removing visibility map drops SR 5.3% / SPL 3.6; removing planner drops SR 6.5% / SPL 2.3; removing object-level semantics reduces SR but raises SPL slightly (32.0 vs 31.6); CLIP outperforms BLIP, matches BLIP-2 while being more efficient.

Limitations

  • Validated only on object navigation in simulation; no real-world robot experiments.
  • Local planner (from VLFM) causes significant degradation in narrow passages across all datasets.
  • MP3D mesh artifacts (holes allowing see-through) cause false-positive target detections, limiting gains on that benchmark.
  • 3D voxel map with 45,000 voxels at 0.25 m resolution requires ~13 GB VRAM on an RTX 4090, which may not scale to very large environments.
  • Belief map currently not extended to manipulation or interaction tasks (acknowledged as future work).

Relevance to Vision-Language Models

BeliefMapNav directly addresses a known failure mode of VLMs in embodied AI: poor spatial reasoning when used as a direct goal-selection oracle. Rather than querying a VLM for a next waypoint, the system grounds CLIP visual-language embeddings into a persistent 3D spatial structure, using language only where it excels (commonsense landmark generation via GPT-4o) while delegating spatial memory to the voxel map. This hybrid architecture—VLM/LLM as semantic prior generator, not spatial reasoner—is a concrete design pattern for integrating foundation models into robotics perception pipelines. The ablation confirming that CLIP and BLIP-2 perform comparably (while BLIP lags) also provides practical encoder selection guidance for retrieval-based navigation tasks.