Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation¶

🕒 Published (v1): 2026-07-08 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LaMem-VLA is a robotic manipulation framework that addresses the temporal short-horizon bias in Vision-Language-Action (VLA) models by introducing dual-scale latent memory—short-term visual and long-term semantic—that is stored, retrieved, and consumed entirely within the VLA's native embedding space. Unlike prior approaches that attach retrieved history as external policy-side context, LaMem-VLA weaves compact latent memory tokens directly into the VLA reasoning sequence before action formation. On LIBERO and SimplerEnv-Bridge, it achieves state-of-the-art success rates, outperforming prior memory-augmented VLA methods.

Problem¶

Mainstream VLA models operate under a Markovian assumption ($a_{t:t+H-1} = \Pi_\theta(o_t, I)$), making action prediction depend solely on the current observation. This causes a temporal short-horizon bias: models cannot track completed subtasks, state transitions, or long-horizon task progress. Existing memory-augmented approaches either (a) expand the observation window (fixed temporal ceiling, growing compute cost) or (b) retrieve from an external memory bank and inject it as auxiliary policy-side context after VLA reasoning—neither integrates memory natively into the VLA's internal latent reasoning process.

Method¶

LaMem-VLA introduces four coordinated modules operating entirely in the VLA's continuous latent space:

Latent Memory Curator: At each timestep, compresses current visual tokens $X_t$ via an SE-bottleneck module into short-term key-value units ($k_s = \text{MeanPool}(C_s(X_t))$, $v_s = C_s(X_t)$) stored in $\mathcal{M}_\text{short}$, and writes action hidden states $H^\text{action}$ sequentially into $\mathcal{M}_\text{long}$. When vault capacity $L=16$ is exceeded, the most redundant adjacent pair (by cosine similarity of keys) is merged by averaging.
Latent Memory Seeker: Builds a context-aware query $Q_t \in \mathbb{R}^{K_q \times C}$ from the current multimodal cognition representation $Q_t^\text{con}$ via a masked-attention transformer query builder. Uses mean-pooled $q_t = \text{MeanPool}(Q_t)$ to retrieve the top-$K=8$ units from each vault by cosine similarity, yielding raw evidence $Z^\text{short}$ and $Z^\text{long}$.
Latent Memory Condenser: Reconstructs retrieved evidence into fixed-length tokens via lightweight transformer memory formers conditioned on $Q_t$: $$M^\text{short} = F_v([Q_t; Z^\text{short}; T_s])_{[-L_s:]}, \quad M^\text{long} = F_c([Q_t; Z^\text{long}; T_l])_{[-L_l:]}$$ producing $L_s=8$ short-term and $L_l=4$ long-term latent memory tokens in the same $C$-dimensional space as the VLA backbone.
Latent Memory Weaver: Prepends both memory token streams (with learnable source embeddings $b_s, b_l$) to the current observation, instruction, and action query tokens before the LLM forward pass: $$S_t = [M^\text{short} + \mathbf{1}_{L_s}b_s^\top;\, M^\text{long} + \mathbf{1}_{L_l}b_l^\top;\, X_t;\, \mathcal{I};\, Q^\text{action}]$$ The resulting memory-grounded action tokens $Z^\text{action}$ condition a diffusion-based action expert (DDIM, 10 steps) to generate 7-DoF action chunks of horizon $H=16$.

The backbone is a 7B-parameter Prismatic VLM pretrained on Open-X Embodiment; the diffusion expert has ~300M parameters. Training uses 8× H800 GPUs with FSDP, global batch size 256, lr $2\times10^{-5}$.

Key Contributions¶

A new paradigm treating robotic history as context-native latent memory: stored, retrieved, and consumed in the same embedding space as VLA perception and reasoning, rather than as external policy-side context.
LaMem-VLA framework with dual memory vaults (short-term visual, long-term semantic) and a capacity-bounded cosine-similarity merge strategy for vault maintenance.
A latent memory condensing mechanism that maps variable-length retrieved evidence into fixed-length memory tokens compatible with VLA reasoning, decoupling memory storage from memory consumption.

Results¶

LIBERO (average across suites): 97.6% success rate vs. MemoryVLA 96.5% (+1.1 pp), CogACT 93.2% (+4.4 pp), π0 94.1% (+3.5 pp on first four suites).
SimplerEnv-Bridge (average): 73.9% vs. CogACT 57.3% (+16.6 pp), π0 69.2% (+4.7 pp), MemoryVLA 71.9% (+2.0 pp), SemanticVLA 65.1% (+8.8 pp).
Per-task on SimplerEnv-Bridge: Spoon on Towel 83.3%, Carrot on Plate 75.0%, Stack Cube 41.7%, Eggplant in Basket 95.8%.

Limitations¶

Empirical validation is exclusively in simulation (LIBERO, SimplerEnv-Bridge); real-world robot experiments are deferred to a future version.
Top-$K$ retrieval from vaults uses a discrete, non-differentiable operation—gradients do not flow through vault selection.
Vault capacity $L=16$ and retrieval $K=8$ are fixed hyperparameters; behavior on very long-horizon tasks exceeding vault capacity is not analyzed.
Single third-person RGB input at 224×224; no depth, wrist camera, or multi-view inputs are evaluated.

Relevance to Vision-Language Models¶

LaMem-VLA directly extends VLM-based architectures (Prismatic/LLaMA-7B backbone) into the action domain by demonstrating that the continuous latent token space of VLMs can natively absorb structured episodic memory without architectural surgery. The core insight—that historical context should participate in the VLM's self-attention over the full token sequence rather than conditioning a downstream head—is transferable to any VLM application requiring temporal grounding (e.g., video QA, multi-turn embodied dialogue). The condenser/weaver design also offers a blueprint for controlled context injection into frozen or finetuned VLMs where context length is bounded. For VLM researchers, this work provides evidence that the latent embedding space is expressive enough to unify perception, language, memory, and action under a single forward pass.