Skip to content

HAMLET: Switch Your Vision-Language-Action Model into a History-Aware Policy

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; HAMLET: history-aware policy for VLA manipulation models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HAMLET is a plug-in fine-tuning framework that adds history-awareness to pre-trained Vision-Language-Action models (VLAs) without costly re-training from scratch. It introduces compact moment tokens initialized via time-contrastive learning and a lightweight Transformer memory module that selectively aggregates these tokens across timesteps. Applied to GR00T N1.5, it achieves a 47.2% absolute improvement on history-dependent real-world tasks.

Problem

State-of-the-art VLAs (GR00T N1.5, CogACT, π0) operate under a single-frame assumption—each action prediction is conditioned only on the current observation. This makes them fundamentally unable to handle non-Markovian tasks where the correct action depends on prior context (e.g., whether an object has already been grasped, which item was acted on first). Naïve remedies—appending raw past frames—increase forward-pass latency by ~35% and peak memory by ~3.6×, degrade generalization due to causal confusion, and reduce feasible batch sizes.

Method

HAMLET augments any pre-trained VLA with two components, both added as lightweight fine-tuning add-ons:

Moment tokens. At each timestep \(t\), a set of \(n_m\) learnable tokens \(m_t \in \mathbb{R}^{n_m \times d}\) is appended to the VLM input. The VLM processes \([o_t, c; m_t]\) and produces updated representations \([h_t; m'_t]\); \(m'_t\) becomes a compact, context-aware summary of timestep \(t\). To make these tokens temporally discriminative, they are pre-initialized with time-contrastive learning: for each anchor timestep \(t\), a positive is an augmented view of the same frame and a hard negative is drawn from a different timestep \(t' \neq t\) in the same trajectory. The objective is: $\(\mathcal{L}_{\text{TCL}}(z_t, z^+_t) = -\sum_{t=1}^{B} \log \frac{\exp(\text{sim}(z_t, z^+_t)/\tau)}{\exp(\text{sim}(z_t, z^+_t)/\tau) + \exp(\text{sim}(z_t, z^-_t)/\tau)}\)$ The VLM backbone \(F_\theta\) is frozen during this stage.

Memory module. A shallow Transformer \(M_\phi\) (default: 2 layers) applies causal self-attention over the stacked history matrix \(M' = [m'_{t-k(T-1)}; \ldots; m'_t] \in \mathbb{R}^{L \times d}\) where \(L = T \cdot n_m\). The last \(n_m\) rows of the output, \(\tilde{m}'_t\), form history-augmented features. These are concatenated with \(h_t\) and fed into the diffusion-based action expert: $\([a_t, \ldots, a_{t+k-1}] = A_\psi([h_t; \tilde{m}'], s_t)\)$ The full pipeline is trained end-to-end with the standard action prediction loss.

Key Contributions

  • Identifies single-frame assumption as a critical bottleneck in current VLAs and proposes HAMLET as a backbone-agnostic, plug-in cure requiring no large-scale re-training.
  • Moment tokens initialized with time-contrastive learning that focus attention on task-relevant, temporally changing regions (gripper, objects) rather than static background.
  • Lightweight causal-Transformer memory module that selectively attends to informative past timesteps, avoiding causal confusion and redundancy introduced by naive frame concatenation.
  • Demonstrates consistent gains across two VLA families (GR00T N1.5, CogACT) and across real-world + three simulation benchmarks.

Results

  • Real-world (GR00T N1.5 backbone): HAMLET achieves 76.4% average success vs. 29.2% for the base model and 45.8% for multi-frame—a +47.2% absolute gain over baseline.
  • Pick-and-Place Twice: 66.7% vs. 12.5% (base), 45.8% (multi-frame)
  • Cover-and-Stack: 79.2% vs. 37.5% (base), 33.3% (multi-frame)
  • Swap Cubes: 83.3% vs. 37.5% (base), 58.3% (multi-frame)
  • RoboCasa Kitchen (100 demos): 65.4% vs. 62.6% (base), 59.3% (multi-frame); at 300 demos: 66.4% vs. 64.1% (base).
  • LIBERO (average across 4 suites): 97.6% vs. 95.6% (base), 86.8% (multi-frame); multi-frame hurts by 8.8pp.
  • SimplerEnv-Bridge (CogACT backbone): 63.5% vs. 52.1% (base), 47.9% (multi-frame)—multi-frame again regresses below baseline.
  • Multi-frame baseline consistently underperforms or regresses vs. the single-frame VLA on simulation benchmarks, confirming causal confusion as a real failure mode.

Limitations

  • Evaluated on a narrow set of real-world tasks (3 tabletop tasks, 24 trials each)—statistical power is limited.
  • History length \(T\) and chunk length \(k\) are fixed hyperparameters; the framework does not learn when to retrieve or discard history adaptively beyond the causal attention window.
  • Time-contrastive pre-initialization adds a separate training stage; interaction with different VLM backbones that use non-causal attention is unexamined.
  • Experiments cover only diffusion/flow-matching action-head VLAs; applicability to token-autoregressive VLAs (e.g., RT-2 style) is not demonstrated.
  • The memory module's fixed history window \(T\) may be insufficient for very long-horizon tasks extending far beyond \(T \cdot k\) timesteps.

Relevance to Vision-Language Models

HAMLET directly targets a structural weakness of VLM-based robot policies: the single-frame bottleneck imposed by standard VLM inference. By treating the VLM as a frozen feature extractor and attaching lightweight learned peripherals (moment tokens + memory Transformer), the work demonstrates that VLM priors can be extended into the temporal domain without retraining the core model—a practically important finding for scaling VLA deployment. The time-contrastive initialization strategy is directly transferable to any VLM application where compact, temporally discriminative representations are needed, such as video understanding or multi-step visual QA. The result that naive multi-frame concatenation hurts performance is a useful empirical warning for VLM practitioners designing temporal context mechanisms.