Turning Adaptation into Assets: Cross-Domain Bridging for Online Vision-Language Navigation¶

🕒 Published (v1): 2026-05-22 05:59 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; test-time adaptation for VLN under non-stationary distribution shifts

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

IDEA (Inter-Domain BridgE with Historical Assets) is a test-time adaptation (TTA) framework for Vision-and-Language Navigation (VLN) that reframes online adaptation as accumulation and reuse of structured knowledge assets rather than isolated per-domain updates. It uses Fisher-guided soft prompt tuning to encode transferable adaptation knowledge, stores these as triplet-structured assets in a dynamic library, and constructs a cross-domain bridge via closed-form convex-hull projection under Wasserstein distance. This eliminates catastrophic forgetting and negative transfer, yielding state-of-the-art results across REVERIE, R2R, and R2R-CE benchmarks.

Problem¶

Existing TTA methods for VLN treat each domain shift as an isolated transfer task, updating a fixed parameter set without preserving or reusing prior adaptation. This causes: (1) catastrophic forgetting — online updates overwrite earlier adaptations when scenes recur; (2) negative transfer — adaptation signals from one domain are blindly applied to dissimilar domains, degrading performance. Feedback-driven methods that use foundation models as oracles are also prohibitively expensive for real-time use.

Method¶

IDEA introduces two synergistic components:

1. Domain Asset Library with Sensitivity-Aware Alignment Learnable soft visual prompts $P := \{p_i\}_{i=1}^L$, $p_i \in \mathbb{R}^C$, are appended to visual tokens $\tilde{V}_t = [P; V_t]$ and optimized via a multi-layer moment-matching loss: $$P^\star = \arg\min_P \sum_{\ell=1}^M \alpha_\ell \, d^{(\ell)}(P), \quad d^{(\ell)}(P) = \|\mu_S^{(\ell)} - \mu_t^{(\ell)}(P)\|_2 + \|\sigma_S^{(\ell)} - \sigma_t^{(\ell)}(P)\|_2$$ Layer weights $\alpha_\ell$ are updated via a Fisher-guided scheme: the trace of the Fisher Information Matrix $\Phi(z) = \mathbb{E}_{a \sim \pi_\theta}[\nabla_z \log \pi_\theta(a)\nabla_z \log \pi_\theta(a)^\top]$ serves as a tractable proxy for Hessian curvature, measuring how policy-sensitive each layer's representations are. Layers with higher Fisher trace get higher $\alpha_\ell$, isolating task-essential priors from domain-specific noise. Each optimized prompt is stored as a triplet asset $\mathcal{A} := \{P^\star, \Gamma, u\}$ — learned prompt, domain statistics, and prediction entropy $u = \mathrm{Ent}(\pi_{\theta, P^\star})$.

2. Cross-Domain Bridge via Convex-Hull Projection Given asset library $\mathcal{M} = \{\mathcal{A}_i\}_{i=1}^K$, the bridge prompt and distribution are linearly interpolated: $\hat{P}_b(w) = \sum_j w_j P_j$, $\Gamma_b(w) = \sum_j w_j \Gamma_j$. Optimal weights $w^\star$ are found by minimizing the 2-Wasserstein distance between target statistics $\Gamma_t$ and $\Gamma_b(w)$, with an uncertainty-aware regularizer penalizing high weights on unreliable assets: $$\min_w W(\Gamma_t, \Gamma_b(w)) + \lambda \sum_j u_j w_j^2, \quad \text{s.t.} \; \mathbf{1}^\top w = 1,\; w \geq 0$$ This is cast as a quadratic program and solved in closed form via KKT conditions: $$w^\star = H^{-1}(g - \nu \mathbf{1}), \quad \nu = \frac{\mathbf{1}^\top H^{-1} g - 1}{\mathbf{1}^\top H^{-1} \mathbf{1}}$$ where $H = A^\top A + \lambda U$, avoiding iterative solvers entirely. At each step, IDEA checks whether the bridge sufficiently covers the current domain ($d_p < \tau \cdot d_0$); if so, it applies the bridge directly (training-free); otherwise it optimizes a new asset and merges or appends it to the library.

Key Contributions¶

First TTA framework for VLN that transforms adaptation into structured, composable, reusable assets — enabling a plug-and-play paradigm instead of isolated per-domain updates.
Fisher-guided layer-weighting scheme for soft prompt optimization, isolating policy-sensitive parameters to improve asset transferability.
Closed-form convex-hull projection bridge construction via KKT conditions, providing training-free adaptation shortcuts without iterative solvers.
Theoretical analysis proving the bridge (a) tightens a generalization error upper bound and (b) is Lipschitz-stable w.r.t. test-time estimation noise.
Demonstrated asset library portability: pre-built libraries can be shared to bypass cold-start for new agents.

Results¶

REVERIE (val unseen): - HAMT + IDEA: SR 34.92% vs. best prior (ReCAP) 33.06% (+1.86%), SPL +1.24% - DUET + IDEA: SR 56.92% vs. best prior 54.74% (+2.18%), RGSPL 39.84% vs. 36.52%

REVERIE (test unseen): - DUET + IDEA: SR 32.81%, SPL 28.52%, RGSPL 14.45% — all best among TTA variants

R2R (val unseen): - DUET + IDEA: SR 76%, SPL 67% vs. FSTTA 75%/62% - BEVBert + IDEA: SR 76%, SPL 68% vs. FSTTA 74%/63%

R2R-CE (val unseen, BEVBert): - IDEA: SR 62%, SPL 52% vs. ReCAP 60%/50%

Average improvement across benchmarks: +2.5% SR and +1.9% SPL over prior TTA methods.

Inference time: IDEA runs at ~245–344 ms/episode vs. FSTTA's 5.49×10³–7.31×10³ ms, comparable to non-iterative methods and far faster than FM-feedback approaches.

Limitations¶

Requires precomputed source-domain feature statistics (mean/std from 128 samples), introducing a light dependency on source data access at deployment.
Asset library capacity $K_{\max}$ is fixed; the nearest-neighbor merging strategy may lose fine-grained domain distinctions as the library saturates.
Soft prompts only modulate visual tokens; language branch and fusion parameters remain frozen, potentially leaving cross-modal alignment incomplete.
Evaluation is confined to indoor 3D navigation benchmarks (Matterport3D environments); generalization to outdoor or highly dynamic scenes is untested.
Theoretical guarantees assume Gaussian feature distributions, which may not hold in highly non-stationary or multi-modal environments.

Relevance to Vision-Language Models¶

IDEA directly addresses a practical deployment gap for VLN models built on vision-language transformers: these models are trained offline but must operate under continuous domain shift at test time. The Fisher-guided prompt tuning strategy is particularly relevant to VLM researchers, as it provides a principled, training-free mechanism to adapt frozen pre-trained cross-modal encoders without modifying weights — a key concern when deploying large VLMs in embodied settings. The convex-hull asset composition paradigm also offers a general recipe for continual, forgetting-free adaptation that could extend to other VLM tasks (e.g., visual question answering under domain shift). This work sits at the intersection of prompt-based VLM adaptation and continual/test-time learning, advancing the understanding of how structured knowledge reuse can replace expensive fine-tuning.