VTM-Nav: Hierarchical Visual-Topological Memory for Cross-Episode Object-Goal Navigation¶

🕒 Published (v1): 2026-07-16 03:11 UTC · Source: Arxiv · link

Why this paper was selected

VLM-driven object-goal navigation with topological memory; training-free embodied agent system

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VTM-Nav introduces a training-free VLM navigation framework that maintains a persistent hierarchical Visual-Topological Memory (VTM) across repeated episodes in the same scene, enabling an embodied agent to reuse accumulated room-connectivity and object-viewpoint experience. Unlike standard episode-isolated ObjectNav, the proposed Cross-Episode ObjectNav setting lets the agent's scene-specific memory persist and grow while model parameters remain fixed. VTM-Nav outperforms WMNav and a cross-episode textual-memory–augmented WMNav baseline on all three tested benchmarks.

Problem¶

Standard ObjectNav benchmarks follow an episode-isolated protocol: all scene knowledge is discarded after each episode, forcing agents to re-explore familiar environments from scratch on every run. This mismatches real deployment, where a robot repeatedly operates in the same home or office and should exploit previously discovered room topology, object viewpoints, and successful approach routes. Existing memory-augmented methods either target within-episode memory or address continual adaptation (changing goal categories); no prior work isolates the value of cross-episode, gradient-free experience reuse under fixed VLM weights.

Method¶

VTM-Nav augments a WMNav-style VLM backbone with a scene-scoped, two-level Visual-Topological Memory \(M_s = (T_s, O_s, H_s)\).

Room-level topology \(T_s = (V_s^{\text{room}}, E_s^{\text{room}})\): a graph of room nodes connected by observed transitions. New nodes are promoted only after repeated tentative support or high-confidence observations to prevent transient noise from polluting the topology.
Object-level memory per room: each room node \(v\) owns in-room nodes \(O_{s,v}^{\text{in}}\) and remote-visible nodes \(O_{s,v}^{\text{rem}}\) (objects glimpsed through doorways). Object-node confidence is updated incrementally; successful approach records receive an additional boost \(\Delta_o^+ > \Delta_o^0\).
Room localization at each step scores candidate room nodes by \(S_{\text{loc}}(v, z_t) = S_{\text{sem}} + S_{\text{topo}} + S_{\text{sp}}\), combining semantic Jaccard overlap, topological adjacency, and spatial proximity.
Cross-episode retrieval: given target \(g\), in-room evidence is scored as \(R^{\text{in}}(v,g) = \gamma_{v,g} + \eta_1 n_{v,g} + \eta_2 n_{v,g}^{\text{succ}}\); BFS over \(T_s\) produces a room-hop path \(\pi_t\) to the best-matching room.
Soft candidate bias: retrieved priors are injected into the VLM prompt as soft context; a deterministic lexical scoring function \(B_{t,k}\) re-ranks navigable candidates using positive (room/object name, approach hint) and negative (wall, closed door) cues, overriding VLM selection only when the bias score clears a confidence threshold.
Conservative execution guard: post-selection heuristic that suppresses no-progress motion, short-horizon oscillations, blocked approaches, and premature stopping, while preserving target-visible and high-confidence actions.

At episode end, room visits, transitions, object observations, and outcomes are written back; successful episodes additionally consolidate target-room associations, terminal viewpoints, and approach cues for future episodes.

Key Contributions¶

Formal definition of Cross-Episode Object-Goal Navigation: same scene, fixed model parameters, self-acquired scene memory evolving across episodes.
VTM-Nav: a training-free VLM framework with a persistent hierarchical Visual-Topological Memory supporting coarse-to-fine (room → object) retrieval.
Visually grounded memory-reuse mechanism: soft memory injection into the VLM prompt plus deterministic lexical candidate re-ranking, biasing only when memory agrees with current observations.
Conservative execution guard reducing oscillations, blocked approaches, revisits, and premature stopping without altering VLM or low-level controller.
Controlled experimental design isolating the contribution of structured visual-topological experience reuse from perception or action differences (identical VLM backbone and action pipeline across all compared methods).

Results¶

VTM-Nav achieves best performance across all three benchmarks (HM3D v0.1, HM3D v0.2, MP3D) under the same-scene cross-episode protocol.
Compared against two baselines sharing the identical VLM backbone and action pipeline:
WMNav (standard episode-isolated)
WMNav + cross-episode textual memory (strengthened baseline augmented with scene-scoped textual experience)
VTM-Nav outperforms both on success rate; the paper does not report specific numeric SR/SPL figures in the provided excerpt but claims consistent improvement across all datasets.

(Note: precise per-benchmark numbers are not included in the provided text excerpt.)

Limitations¶

Memory is built solely from egocentric observations and estimated poses; no ground-truth maps, oracle room labels, or shortest paths are used, so localization noise can corrupt the stored topology.
Room localization relies on VLM-generated room-category labels, which can be inconsistent or noisy, requiring conservative node-promotion logic to avoid pollution.
Candidate re-ranking uses simple lexical matching rather than semantic embeddings, limiting robustness to paraphrase or description variability.
Cross-episode memory is scene-specific (\(M_s\) cannot be transferred to scene \(s' \neq s\)), so the approach does not generalize to novel scenes without fresh accumulation.
Evaluation is restricted to same-scene repeated episodes; generalization to multi-scene lifelong settings is not demonstrated.
Exact quantitative improvements over baselines are not fully detailed in the provided excerpt, making it hard to assess effect size.

Relevance to Vision-Language Models¶

VTM-Nav exemplifies a growing paradigm in which VLMs serve as frozen, zero-shot reasoners within embodied agents, and the research challenge shifts to what persistent structured state should complement them. The paper provides a concrete architecture for coupling VLM open-vocabulary semantics with a topological memory that survives episode resets, directly addressing the deployment gap for VLM-based navigation agents. For researchers tracking VLMs, the key design insight is that soft memory injection into VLM prompts — rather than fine-tuning or hard overrides — preserves the generality of the underlying model while enabling experience reuse, a pattern transferable to other VLM-agent settings. The cross-episode evaluation protocol also represents a more realistic testbed for VLM agents, pushing the community toward deployment-oriented benchmarks beyond isolated episodes.