Skip to content

Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dynam3D is a 3D-VLM for monocular vision-and-language navigation that builds dynamic, hierarchical patch-instance-zone representations from posed RGB-D input by lifting CLIP features into 3D space and updating them online via frustum culling and a learned Merging Discriminator. It replaces the video-frame paradigm used by prior VLN systems with structured, language-aligned 3D scene tokens fed into a 3.8B LLaVA-Phi-3-mini backbone. The system sets new state-of-the-art on R2R-CE, REVERIE-CE, and NavRAG-CE under the monocular setting and transfers to real-world robot deployment.

Problem

Video-based VLMs applied to monocular VLN suffer three compounding failures: (1) they cannot capture fine-grained 3D geometry and spatial semantics, hampering large-scale exploration and error correction; (2) they lack structured scene memory, precluding pre-exploration and lifelong learning; (3) they are brittle to dynamic environments where objects or people move, because historical frames become stale with no principled update mechanism.

Method

3D Feature Extraction. CLIP-ViT-L/14@336px encodes RGB frames into 576 patch features (24×24), which are back-projected into 3D world coordinates using depth maps and camera intrinsics/extrinsics to form a growing feature-point set \(M\).

Frustum Culling Update. When a new RGB-D frame arrives, existing feature points whose 3D coordinates project inside the current camera frustum at a depth shallower than the observed surface are discarded; new patch features are then added. This removes moved or occluded objects without touching regions outside the current field-of-view (which serve as long-term memory).

Instance Encoding & Merging. FastSAM segments the RGB image into 2D instance masks. A transformer-based InstanceEncoder aggregates patch features within each mask (with 3D relative-position embeddings) into a compact instance vector via a learnable query token. A Merging Discriminator MLP evaluates candidate 2D→3D instance pairs using semantic similarity, 3D instance features, and Euclidean distance to decide whether to merge or create a new 3D instance, trained on ScanNet/HM3D/Matterport3D/3RScan with binary classification loss.

Zone Encoding. 3D space is partitioned into uniform cubic zones; a Zone Encoder aggregates 3D instance features within each zone into a coarse spatial-layout representation capturing room-level context (kitchen, bedroom, etc.).

CLIP Alignment. Instance and zone representations are aligned to CLIP's semantic space via contrastive learning on SceneVerse/g3D-LF 3D-language pairs. To mitigate per-view feature bias (ground-truth 2D CLIP averages differ across viewpoints), Subspace Contrastive Learning subtracts the view-mean (semantic center \(V_j\)) before computing cosine similarity, anchoring contrastive learning in a shared subspace rather than the global CLIP origin.

3D-VLM. A generalizable feature field renders panoramic 3D patch tokens on 12×48 rays (90° vertical × 360° horizontal) around the agent. These patch tokens, plus instance tokens and zone tokens (sorted by distance), are concatenated with instruction and 4-step history tokens and fed into LLaVA-Phi-3-mini (3.8B) for autoregressive action prediction (turn θ°, forward d cm, stop).

Key Contributions

  • Dynamic patch-instance-zone 3D representation with online encoding, Frustum Culling, and hierarchical layer-wise updates for real-time scene adaptation.
  • Merging Discriminator for multi-view geometrically consistent 3D instance tracking across frames.
  • Subspace Contrastive Learning that mitigates cross-view feature bias in CLIP distillation for 3D instance supervision.
  • 3D-VLM integrating generalizable feature-field panoramic patches with instance and zone tokens, enabling action prediction at 3.8B parameters (outperforming 7B video baselines).
  • Demonstrated pre-exploration and lifelong memory modes; validated on a Hello Robot Stretch 3 in static and dynamic real-world environments.

Results

  • R2R-CE Val SR: 52.9% vs. g3D-LF 47.2% (+5.7%) and Uni-NaVid 47.0%; Test SR 51.4% vs. g3D-LF 46.3%. SPL 45.7 vs. g3D-LF 34.6.
  • REVERIE-CE Val SR: 40.1% vs. NaVid 26.6% (+13.5%) and g3D-LF 34.4%.
  • NavRAG-CE Val SR: 24.7% vs. NaVid 19.4% (+5.3%) and g3D-LF 21.4%.
  • Pre-exploration on R2R-CE / REVERIE-CE: SR 57.1% / 48.1% (+4.2pp / +8.0pp over base Dynam3D).
  • Lifelong Memory on R2R-CE / REVERIE-CE: SR 55.6% / 45.0%.
  • Both on R2R-CE / REVERIE-CE: SR 58.4% / 49.5% (>20pp above NaVid on both).
  • Real-world static: SR 55% vs. NaVid 35%, g3D-LF 30%; +pre-exploration: 70%.
  • Real-world dynamic: SR 45% vs. NaVid 20%, g3D-LF 10%; +pre-exploration: 45%.
  • Noise robustness: simultaneous SLAM+depth noise reduces SR by only ~2% (52.9% → 50.9%).
  • Inference: 649 ms/step on RTX 4090 (83 ms for 3D updates, 540 ms for LLM).

Limitations

  • No explicit target-coordinate output, limiting applicability to mobile manipulation tasks requiring precise localization.
  • No question-answering, dialogue, or dynamic task-update capabilities.
  • 3D-language pretraining data is orders of magnitude smaller than 2D image-language corpora (millions vs. billions), constraining 3D representation generalization.
  • Evaluated only in indoor environments (Matterport3D-derived); outdoor or unstructured scene generalization is untested.

Relevance to Vision-Language Models

Dynam3D directly addresses one of the hardest open problems for VLMs in embodied settings: how to ground language understanding in persistent, geometrically accurate 3D scene representations rather than raw pixel sequences. By showing that a 3.8B VLM with structured 3D tokens outperforms 7B video-VLMs on navigation, it challenges the assumption that scale in 2D video pretraining is the right axis for embodied reasoning. The Subspace Contrastive Learning and Merging Discriminator techniques are broadly applicable primitives for lifting any 2D CLIP-aligned VLM into 3D-grounded operation. For researchers tracking VLMs, this represents a concrete architecture for scene-persistent, language-aligned 3D perception that extends the VLM paradigm from passive understanding to active navigation.