Uncertainty-Aware Gaussian Map for Vision-Language Navigation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; uncertainty-aware Gaussian map improves VLN in partially observed 3D environments

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Vision-Language Navigation (VLN) agents traditionally ignore perceptual uncertainty when selecting actions, leading to failures under ambiguous visual conditions. This work explicitly models three forms of perceptual uncertainty—geometric, semantic, and appearance—within a 3D Gaussian Splatting scene representation, fusing them into a unified 3D Value Map that informs navigation decisions. Evaluations on R2R, RxR, and REVERIE benchmarks show consistent improvements over strong baselines.

Problem¶

Existing VLN agents commit to action predictions regardless of perceptual confidence. When facing visually similar structures (e.g., multiple identical doors), occlusions, or texture ambiguity, agents have no mechanism to represent or exploit unreliable percepts, leading to suboptimal or unsafe navigation. Prior uncertainty work in VLN (e.g., VLN-Copilot) operates at the decision level only, not at the perceptual level where the uncertainty originates.

Method¶

The agent builds a Semantic Gaussian Map (SGM) at each waypoint from panoramic RGB-D observations. Pixels are back-projected into 3D via depth and camera intrinsics (Eq. 1), initializing Gaussian primitives \(g_i\) with parameters \(\{\mu_i, \Sigma_i, \alpha_i, c_i, s_i\}\). Semantic attributes \(s_i \in \mathbb{R}^3\) are CLIP embeddings of SAM2 segments, attached per primitive. The SGM is optimized by differentiable alpha-blending rendering (Eqs. 2–3) with L1 + SSIM losses on color, depth, and semantics.

Three uncertainty types are then estimated:

Geometric uncertainty (\(U_i^g\)): Position \(\mu_i\) and scale \(e_i\) are treated as random variables with learnable perturbations \(\chi_i^\mu, \chi_i^e\). A variational posterior \(q_\phi(\chi)\) is learned by maximizing the ELBO against a zero-mean Gaussian prior on position and a scale-proportional uniform prior on scale. \(U_i^g\) is the L2 norm of extracted standard deviations of both variational distributions (Eq. 6).
Semantic uncertainty (\(U_i^s\)): Semantic attribute \(s_i\) is perturbed by a learnable offset \(\chi_i^s\), with the same variational inference framework using a \(\mathcal{N}(0, \epsilon^2 I)\) prior. \(U_i^s\) is the L2 norm of the posterior standard deviation (Eq. 7).
Appearance uncertainty (\(U_i^a\)): Defined as the log-determinant of the per-Gaussian block of the Fisher Information matrix of the reconstruction loss \(L_r = \frac{1}{2}\|\hat{I}-I\|_2^2\) with respect to all Gaussian parameters. In a refined SGM the Hessian reduces to the Fisher Information; the log-determinant gives a scalar volume of the uncertainty ellipsoid in parameter space (Eq. 9).

Each Gaussian is augmented to \(g_i \in \mathbb{R}^{20}\) with \(\{U_i^g, U_i^s, U_i^a\}\), forming the 3D Value Map. Gaussian features are projected to \(\mathbb{R}^{768}\), pooled into \(F^g\), concatenated with instruction embedding \(X\), and fed to a multi-layer transformer to predict waypoint probabilities (Eqs. 11–12). A dynamic topological memory graph stores both 2D panoramic and 3D Value Map features across timesteps.

Key Contributions¶

Explicit formulation of three perceptual uncertainty types (geometric, semantic, appearance) within a 3DGS scene representation for VLN.
Variational inference framework for geometric and semantic uncertainty estimation over Gaussian position, scale, and semantic attributes.
Fisher Information-based appearance uncertainty as a tractable Hessian proxy, computed per-Gaussian via block-diagonal approximation.
Unified 3D Value Map that grounds uncertainty as navigation affordances and constraints, integrated into a transformer-based action predictor.
State-of-the-art or competitive results on R2R, RxR, and REVERIE without task-specific architectural changes per benchmark.

Results¶

R2R (val unseen): +2% SR, +1% SPL over prior best (VER baseline implied).

RxR (multilingual): +1.1% SR, +1.7% nDTW with comparable SDTW over prior best.

REVERIE (val unseen vs. VER): - RGS: 37.65 vs. 33.71 (+2.94%) - RGSPL: 27.01 vs. 23.70 (+2.57%) - SR: 56.37 vs. 55.98 (+0.39%), SPL: 37.64 vs. 39.66 (−2.02%; VER leads on SPL val unseen)

REVERIE (test unseen vs. 3DGS-VLN, closest prior 3DGS-based method): - SR: 55.90 vs. 52.93, RGS: 35.68 vs. 35.65, RGSPL: 25.50 vs. 25.76 (marginal)

All reported figures include ±standard deviation over multiple runs.

Limitations¶

Runtime overhead from SGM construction (SAM2 segmentation + uncertainty estimation) at each navigation step; partially mitigated by offline pretraining but still present during inference.
Fisher Information block-diagonal approximation ignores cross-Gaussian parameter correlations, potentially underestimating global appearance uncertainty.
SAM2 replacement with lightweight variants trades segmentation quality for speed, introducing a quality-efficiency tradeoff with unquantified impact on navigation.
Evaluation is confined to the Matterport3D simulator; real-world generalization is undemonstrated.
SPL on REVERIE val unseen (37.64) does not surpass VER (39.66), suggesting the longer trajectories from uncertainty-driven exploration may penalize path efficiency.

Relevance to Vision-Language Models¶

This paper directly advances grounded VLN—a core embodied VLM task requiring joint language instruction following and 3D spatial reasoning—by showing that explicitly modeling perception-level uncertainty improves action reliability. The use of CLIP embeddings as semantic Gaussian attributes tightly couples a foundational VLM with a 3D scene representation, making the approach a concrete example of how VLM features can be anchored in physically interpretable 3D structures rather than flat token spaces. The 3D Value Map framing—turning VLM-derived semantic features into spatially grounded affordance and reliability signals—is relevant to a growing line of work on 3DGS-based scene understanding for embodied agents. Researchers tracking VLMs in navigation contexts will find the uncertainty quantification methodology transferable to other settings where VLM grounding confidence is task-critical (e.g., open-vocabulary manipulation, robot exploration).