VTM-Nav: Hierarchical Visual-Topological Memory for Cross-Episode Object-Goal Navigation¶

🕒 Published (v1): 2026-07-16 03:11 UTC · Source: Arxiv · link

Why this paper was selected

Training-free VLM-based embodied navigation with hierarchical visual-topological memory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VTM-Nav introduces a training-free VLM navigation framework that maintains a persistent hierarchical Visual-Topological Memory (VTM) across repeated episodes in the same scene, enabling an embodied agent to reuse accumulated room-connectivity and object-viewpoint experience. Unlike standard episode-isolated ObjectNav, the proposed Cross-Episode ObjectNav setting lets the agent's scene-specific memory persist and grow while model parameters remain fixed. VTM-Nav outperforms WMNav and a cross-episode textual-memory–augmented WMNav baseline on all three tested benchmarks.

Problem¶

Standard ObjectNav benchmarks follow an episode-isolated protocol: all scene knowledge is discarded after each episode, forcing agents to re-explore familiar environments from scratch on every run. This mismatches real deployment, where a robot repeatedly operates in the same home or office and should exploit previously discovered room topology, object viewpoints, and successful approach routes. Existing memory-augmented methods either target within-episode memory or address continual adaptation (changing goal categories); no prior work isolates the value of cross-episode, gradient-free experience reuse under fixed VLM weights.

Method¶

VTM-Nav augments a WMNav-style VLM backbone with a scene-scoped, two-level Visual-Topological Memory \(M_s = (T_s, O_s, H_s)\).

Room-level topology \(T_s = (V_s^{\text{room}}, E_s^{\text{room}})\): a graph of room nodes connected by observed transitions. New nodes are promoted only after repeated tentative support or high-confidence observations to prevent transient noise from polluting the topology.
Object-level memory per room: each room node \(v\) owns in-room nodes \(O_{s,v}^{\text{in}}\) and remote-visible nodes \(O_{s,v}^{\text{rem}}\) (objects glimpsed through doorways). Object-node confidence is updated incrementally; successful approach records receive an additional boost \(\Delta_o^+ > \Delta_o^0\).
Room localization at each step scores candidate room nodes by \(S_{\text{loc}}(v, z_t) = S_{\text{sem}} + S_{\text{topo}} + S_{\text{sp}}\), combining semantic Jaccard overlap, topological adjacency, and spatial proximity.
Cross-episode retrieval: given target \(g\), in-room evidence is scored as \(R^{\text{in}}(v,g) = \gamma_{v,g} + \eta_1 n_{v,g} + \eta_2 n_{v,g}^{\text{succ}}\); BFS over \(T_s\) produces a room-hop path \(\pi_t\) to the best-matching room.
Soft candidate bias: retrieved priors are injected into the VLM prompt as soft context; a deterministic lexical scoring function \(B_{t,k}\) re-ranks navigable candidates using positive (room/object name, approach hint) and negative (wall, closed door) cues, overriding VLM selection only when the bias score clears a confidence threshold.
Conservative execution guard: post-selection heuristic that suppresses no-progress motion, short-horizon oscillations, blocked approaches, and premature stopping, while preserving target-visible and high-confidence actions.

At episode end, room visits, transitions, object observations, and outcomes are written back; successful episodes additionally consolidate target-room associations, terminal viewpoints, and approach cues for future episodes.

Key Contributions¶

Formal definition of Cross-Episode Object-Goal Navigation: same scene, fixed model parameters, self-acquired scene memory evolving across episodes.
VTM-Nav: a training-free VLM framework with a persistent hierarchical Visual-Topological Memory supporting coarse-to-fine (room → object) retrieval.
Visually grounded memory-reuse mechanism: soft memory injection into the VLM prompt plus deterministic lexical candidate re-ranking, biasing only when memory agrees with current observations.
Conservative execution guard reducing oscillations, blocked approaches, revisits, and premature stopping without altering VLM or low-level controller.
Controlled experimental design isolating the contribution of structured visual-topological experience reuse from perception or action differences (identical VLM backbone and action pipeline across all compared methods).

Results¶

VTM-Nav achieves best performance across all three benchmarks (HM3D v0.1, HM3D v0.2, MP3D) under the same-scene cross-episode protocol.
Compared against two baselines sharing the identical VLM backbone and action pipeline:
WMNav (standard episode-isolated)
WMNav + cross-episode textual memory (strengthened baseline augmented with scene-scoped textual experience)
VTM-Nav outperforms both on success rate; the paper does not report specific numeric SR/SPL figures in the provided excerpt but claims consistent improvement across all datasets.

(Note: precise per-benchmark numbers are not included in the provided text excerpt.)

Limitations¶

Memory is built solely from egocentric observations and estimated poses; no ground-truth maps, oracle room labels, or shortest paths are used, so localization noise can corrupt the stored topology.
Room localization relies on VLM-generated room-category labels, which can be inconsistent or noisy, requiring conservative node-promotion logic to avoid pollution.
Candidate re-ranking uses simple lexical matching rather than semantic embeddings, limiting robustness to paraphrase or description variability.
Cross-episode memory is scene-specific (\(M_s\) cannot be transferred to scene \(s' \neq s\)), so the approach does not generalize to novel scenes without fresh accumulation.
Evaluation is restricted to same-scene repeated episodes; generalization to multi-scene lifelong settings is not demonstrated.
Exact quantitative improvements over baselines are not fully detailed in the provided excerpt, making it hard to assess effect size.

Relevance to Agentic AI / LLM Agents¶

VTM-Nav directly addresses one of the core unsolved problems in agent harness design: how to give an agent structured, reusable memory that accumulates across deployments without retraining. The cross-episode setting maps cleanly onto real agent scenarios where the same agent repeatedly operates in a known environment (home robot, persistent workspace assistant), and the hierarchical topological memory — storing room connectivity and object viewpoints as a growing graph — is a concrete architectural pattern for grounding VLM reasoning in accumulated spatial experience. The training-free approach is particularly actionable: it shows that a frozen VLM backbone can serve as the reasoning engine while all scene-specific knowledge lives entirely in an external, inspectable memory structure, which is exactly the separation of concerns you want when building maintainable agent harnesses. The key transferable idea is the graph-structured episodic memory that indexes multimodal observations at multiple granularities, a design applicable to any long-horizon agent that needs to avoid re-exploring already-known state.