Embodied Navigation Foundation Model¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; foundation model for embodied navigation leveraging VLM priors

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

NavFoM is a cross-embodiment, cross-task navigation foundation model trained on 12.7M samples (8.02M navigation + 4.76M image/video QA) spanning quadrupeds, drones, wheeled robots, and vehicles across tasks including VLN, object search, target tracking, and autonomous driving. It introduces Temporal-Viewpoint Indicator (TVI) tokens and Budget-Aware Temporal Sampling (BATS) within a unified VLM-based architecture to handle arbitrary camera configurations and token budget constraints. Without task-specific fine-tuning, it achieves state-of-the-art or competitive performance on seven public benchmarks.

Problem¶

Existing VLM-based navigation methods are confined to narrow task domains, embodiment-specific architectures, and fixed camera configurations. No prior work unifies diverse navigation tasks (VLN, object goal navigation, active tracking, autonomous driving) across heterogeneous embodiments under a single model — preventing the kind of broad zero-shot generalization VLMs have achieved in perception tasks.

Method¶

NavFoM extends a video-based VLM (Qwen2-7B LLM backbone with DINOv2 + SigLIP vision encoders and a 2-layer MLP cross-modal projector) with two core components:

Temporal-Viewpoint Indicator (TVI) tokens. Each visual token group is prefixed by a learnable indicator $E^{TVI} \in \mathbb{R}^C$: $$E^{TVI} = E_{Base} + P_{time}(\text{TimePE}(t)) + P_{angle}(\text{AnglePE}(\phi))$$ where $\phi$ is the camera azimuthal angle encoded via $[\sin\phi, \cos\phi]$ sinusoidal embeddings (preserving circular continuity), and $t$ is a sinusoidal temporal index. For video QA, the angle term is dropped; for image QA, only $E_{Base}$ is used. This allows a single token organization scheme to handle all modalities.

Budget-Aware Temporal Sampling (BATS). When accumulated frame tokens exceed budget $B_{token}$, each frame at timestep $t$ is sampled with probability: $$P(t) = (1-\epsilon)e^{k(t-T)/T} + \epsilon, \quad k > 0$$ inspired by the forgetting curve, biasing retention toward recent frames. The exponential rate $k$ is solved offline via Brent's method to satisfy $E[\text{frames}] \leq \frac{B_{token} - (64+1)N}{(4+1)N}$.

Visual tokens use two resolutions: fine-grained ($64$ patches, for the most recent frame and image QA) and coarse-grained ($4$ patches, for navigation history and video QA). Navigation produces waypoint trajectories $\tau = \{a_1,\dots,a_8\}$, $a \in \mathbb{R}^4 = (x,y,z,\theta)$, from a 3-layer MLP planning head. Training co-tunes navigation (MSE loss) and QA (cross-entropy) with loss $\mathcal{L} = \beta \mathcal{L}_{nav} + \mathcal{L}_{QA}$, $\beta=10$.

Key Contributions¶

NavFoM: first cross-embodiment ($\times4$ robot types) and cross-task ($\times4$ task families) navigation foundation model, trained on 12.7M samples.
TVI tokens: a composable indicator embedding scheme enabling a single LLM to distinguish camera viewpoint and temporal order across image QA, video QA, and multi-camera navigation.
BATS: forgetting-curve-inspired frame sampling that adapts to variable camera counts and token budgets at inference time with no retraining.
Zero-shot SOTA on 7 public navigation benchmarks across embodiments without task-specific fine-tuning.
Real-world validation on humanoid robots, quadrupeds, drones, and wheeled robots at 5 Hz (218 ms/step, 19.1 GB VRAM on RTX 4090).

Results¶

VLN-CE RxR (multi-view, 4 RGB cameras): 64.4% SR vs. 56.3% prior SOTA (RGB-D + odometry methods).
VLN-CE RxR (single-view): 57.4% SR vs. 51.8% prior SOTA.
VLN-CE R2R: multi-view adds +5.5% SR over single-view; achieves SOTA across settings.
HM3D-OVON (object search, zero-shot): 45.2% SR vs. 43.6% fine-tuned prior SOTA.
EVT-Bench (active visual tracking): competitive SOTA performance reported in Figure 1 (~49.3% vs ~48.7% for nearest baseline).
NavSim (autonomous driving): competitive PDM-score vs. AD-specific baselines (Figure 1: 62.0% vs ~57.6% nearest baseline).
OpenUAV: SOTA on UAV VLN benchmark.
Real-world: successful deployment across all four embodiment types.

Limitations¶

BATS can become analytically unsolvable for very long sequences (e.g., $T=1120$ under 4-camera setup with $B_{token}=2048$); authors note this is rare in practice.
The trajectory normalization scaling factor $\alpha_{task}$ is task-specific and must be manually set for new embodiment categories.
The loss weighting $\beta=10$ is a fixed heuristic; the paper acknowledges adaptive weighting is an open problem.
Training requires 4,032 H100 GPU-hours (56 GPUs × 72 hours), limiting accessibility.
Evaluation relies on public benchmark splits; the degree of data contamination from pseudo web-video data (Sekai dataset) is not fully analyzed.

Relevance to Vision-Language Models¶

NavFoM demonstrates that VLM architectures — LLM backbone, cross-modal projector, and visual encoders — can be extended to embodied navigation by augmenting token organization rather than redesigning the architecture. The TVI token design is a direct contribution to the broader VLM literature on modality and positional token engineering, complementing work on time/task indicator tokens in multimodal models. Co-training navigation trajectories with image/video QA data shows that navigation benefits from general VLM pretraining, a pattern relevant to any researcher studying how VLMs generalize to action-prediction settings. The BATS strategy addresses token-budget constraints that arise whenever VLMs process long video sequences — a practical concern shared across video understanding, long-context VLMs, and embodied AI applications.