Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-Language Navigation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; dual-system slow-grounding/fast-moving VLM for generalizable VLN

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DualVLN is the first dual-system foundation model for vision-language navigation (VLN) that decouples high-level reasoning from low-level control: a 7B VLM (System 2) slowly grounds mid-term waypoints as 2D pixel coordinates at 2 Hz, while a lightweight diffusion transformer policy (System 1) executes smooth trajectories at 30 Hz. The asynchronous design eliminates the fragmented motion and high latency of end-to-end VLA approaches and achieves state-of-the-art on VLN-CE and VLN-PE benchmarks, with strong real-world generalization across wheeled, quadruped, and humanoid robots.

Problem¶

Existing VLA models for continuous VLN adopt a tightly-coupled end-to-end paradigm that maps vision-language inputs directly to short-horizon discrete actions (e.g., move forward 0.25 m). This causes (1) fragmented, unnatural motion with high inference latency due to per-step calls to large VLMs, (2) inability to achieve high-frequency control needed for dynamic obstacle avoidance, and (3) entanglement of global planning and local control that prevents hierarchical specialization. No prior VLN benchmark evaluates social/dynamic obstacle avoidance.

Method¶

Dual-System Architecture (DualVLN):

System 2 (slow, 2 Hz): Fine-tunes Qwen-VL-2.5 (7B) to perform farthest pixel goal grounding — predicting a 2D pixel coordinate \((u, v)\) in the egocentric image corresponding to the next preferred navigation waypoint. A self-directed view adjustment module uses discrete turn/look actions to seek informative perspectives before committing to a pixel goal. Training samples are generated by projecting 3D VLN-CE trajectories onto 2D images, filtering occluded points via depth comparison.
Latent Goal Bridge: After System 2 generates a pixel goal, four learnable latent queries \(Z\) are appended to the VLM's context sequence \(X\) (instruction + history images + pixel goal) and trained via prompt tuning. The VLM processes \([X; Z]\), producing \(Z'\) — a compact latent representation that captures task-relevant semantic information from the VLM's hidden states for System 1.
System 1 (fast, 30 Hz): A compact Diffusion Transformer (DiT; hidden dim 384, 12 layers, 6 heads) conditioned on (a) the low-frequency latent goal \(Z'\) from System 2 and (b) high-frequency RGB inputs. RGB at time \(t\) (System 2's last frame) and current time \(t+k\) are encoded by a ViT (DepthAnythingV2-Small backbone), fused via self-attention, and compressed to 32 tokens by a Q-Former. The DiT generates 32 dense waypoints per inference step via flow matching, minimizing \(\mathcal{L}_\text{flow} = \mathbb{E}_{u, X_0, \epsilon} \|\hat{\dot{X}}_u - \dot{X}_u\|_2^2\).
Decoupled Sequential Training: System 2 is trained first on multi-source VLN data; its weights are then frozen. System 1 is trained separately on low-level goal-reaching data with latent queries as the only interface.

Social-VLN Benchmark: A new dynamic benchmark built on R2R-CE with humanoid agents placed strategically along ground-truth trajectories (Habitat 3.0). Introduces Human Collision Rate (HCR) metric. Training data: 763K social navigation episodes across 60 MP3D scenes using modified A* replanning triggered by human detection.

Key Contributions¶

First asynchronous dual-system VLN foundation model decoupling VLM-based global planning from diffusion-policy local control
Farthest pixel goal grounding formulation that converts VLN into spatially-interpretable 2D coordinate prediction within Qwen-VL-2.5
Latent goal bridge via prompt-tuned queries \(Z'\), providing richer conditioning for System 1 beyond explicit pixel goals alone
Multi-modal conditioning DiT (System 1) that handles temporal asynchrony between 2 Hz and 30 Hz systems using dual-frame ViT + Q-Former compression
Social-VLN benchmark with HCR metric and 763K dynamic training episodes for evaluating socially-aware navigation

Results¶

VLN-CE R2R Val-Unseen (single-view RGB methods): - DualVLN: SR 64.3%, SPL 58.5%, NE 4.05 m - StreamVLN (prior SOTA): SR 56.9%, SPL 51.9%, NE 4.98 m - NaVILA: SR 47.0%, SPL 42.7%, NE 5.22 m - NaVid: SR 37.4%, SPL 35.9%, NE 5.47 m

VLN-CE RxR Val-Unseen: - DualVLN: nDTW 70.0 - StreamVLN: nDTW 61.9

VLN-PE (zero-shot transfer, humanoid Unitree H1, Val-Unseen): - DualVLN: SR 51.60%, SPL 42.49%, NE 4.66 m - NaVid (best prior zero-shot): SR 22.42%, SPL 18.58%, NE 5.94 m - Methods trained on VLN-PE (CMA+): SR 22.12%, SPL 18.65%

Social-VLN (R2R Val-Unseen): - DualVLN: SR 37.2%, SPL 35.8%, HCR 35.4% - StreamVLN: SR 31.4%, SPL 29.1%, HCR 36.4% - Both drop ~26–27% SR vs. static VLN

Real-World (20 trials/scenario per model): - DualVLN achieves 100% SR in hallway (easy), 85% in bedroom (medium), 95% in R2R office (hard) - All VLM baselines (NaVid, NaVILA, StreamVLN) reach 0% SR on the hard office scenario

Limitations¶

Social-VLN results show ~27% SR degradation under dynamic conditions, indicating substantial room for improvement in socially-aware navigation
System 2 inference latency (0.7 s with KV-cache, 1.1 s without) limits responsiveness; System 1 must tolerate stale latent goals during the gap
System 1 requires high-frequency RGB at 30 Hz but runs full inference on a remote server with an RTX 4090 GPU (20 GB VRAM), raising on-device deployment concerns
Ablation study results are truncated in the provided text; full analysis of individual component contributions is not available here
Evaluated only in Matterport3D-derived environments; outdoor or non-indoor generalization is untested in simulation

Relevance to Vision-Language Models¶

DualVLN directly addresses a core tension in deploying large VLMs for embodied tasks: VLMs excel at semantic reasoning but are too slow and coarse for real-time control. The pixel goal grounding formulation is a concrete adaptation of spatial grounding capabilities (as in Qwen-VL-2.5) to the navigation domain, showing how VLM spatial understanding can be retargeted without destroying pre-trained generalization. The latent query prompt-tuning bridge demonstrates a general technique for extracting task-specific representations from frozen VLMs while preserving their generalization, which is broadly relevant to the VLM-as-planner paradigm in robotics. The work also sets a new trajectory for VLN benchmarking by introducing dynamic social scenarios, pushing VLM-based navigation evaluation beyond static instruction following.