Present but Not Remembered: Auditing How Frozen VLAs Encode, Deploy, and Steer Visual History¶

🕒 Published (v1): 2026-07-03 14:27 UTC · Source: Arxiv · link

Why this paper was selected

Mechanistic audit of how frozen VLAs encode visual history — informs VLA memory architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Frozen vision-language-action models (VLAs) encode past frames linearly at every layer, yet the information unique to history beyond the current frame is near-zero ($A_4/A_1 \lesssim 0.05$) across three models and both architecture families. History is only causally deployed into the action under near-total current-frame destruction, and even then the readout severs its dependence on history by the network midpoint (L6). Whether injected history can steer the action is not determined by encoding but by a fallback-vs-standing deployment regime that is architecture-conditional.

Problem¶

The memory-augmentation literature for VLAs (retrieval buffers, recurrent latents, trajectory overlays) presupposes that native multi-frame policies underuse temporal history, but no prior work had mechanistically audited what a frozen VLA already does with the history it receives. Without that baseline, memory additions are fixes without a characterized fault. The only prior cross-architecture mechanistic VLA study (Grant et al., 2026) fixed the time axis and examined the within-frame vision-language fusion axis; the orthogonal temporal coordinate was unexamined.

Method¶

The authors run a Temporal-Deployment Audit (TDA, Algorithm 1) on frozen, open-loop checkpoints—Octo-Small (27 M), Octo-Base (93 M), and CronusVLA-0.5B—using a content-hashed set of 250 matched Bridge-dataset pairs (no weight updates, single GPU).

Decode leg (what is present vs. unique): A ridge probe $g_\ell$ regresses the previous-frame scene state $s_{t-1}$ on pooled history-token activations $\phi^{(\ell)}$: $$A_1(\ell) = R^2\!\left(g_\ell(\phi^{(\ell)}),\, s_{t-1}\right)$$ To isolate information unique to the past, the target is residualized against what the current frame already explains ($\hat{s}_{t-1|t}$): $$A_4(\ell) = R^2\!\left(g_\ell(\phi^{(\ell)}),\, s_{t-1} - \hat{s}_{t-1|t}\right)$$ 81 probe configurations (layer × pooling × target × regularization) establish the $A_4$ ceiling.

Deploy leg (causal use): At each layer $\ell$, history-token activations from a donor episode $d$ overwrite those of $x$; the corrected contribution subtracts a self-swap null floor ($d = x$): $$\Delta_\ell = C_\ell^{\text{occ\_black}} - C_\ell^{\text{markov}}, \quad C_\ell = C_\ell^{\text{raw}} - C_\ell^{\text{null}}$$ A degradation ladder (partial occlusion → heavy → blur → full black) tests dose-response. A second, mechanistically orthogonal attention-knockout intervention (masking readout-token attention to history keys) confirms the deploy-band cutoff. Effects are hardened via paired bootstrap CIs, 5-seed sign-consistency, and 5,000-resample permutation tests. An injection leg tests whether supplying clean $o_{t-1}$ history at $\ell < \ell^\star$ under occlusion causally repairs the action (the injectability/content-bearing gate).

Key Contributions¶

First mechanistic audit of the temporal axis in VLAs: introduces $A_4$ (history-unique decode) as the temporal complement to Grant et al.'s fusion-axis metric $A_1$, across 81 probe configurations and three frozen models.
Three-layer dissociation: history is (i) linearly decodable at every depth ($A_1$ peaks at $R^2 = 0.389$), (ii) almost entirely redundant with the present ($A_4 \leq 0.019$, i.e., $A_4/A_1 \lesssim 0.05$), and (iii) causally deployed only under near-total current-frame loss, with the readout severed by L6.
Reusable, training-free TDA (Algorithm 1): a drop-in audit for any frozen multi-frame VLA that returns the full profile $(A_1, A_4, \{\Delta_\ell\}, \ell^\star, \sigma, \text{CONTENT-BEARING})$.
Architecture-conditional steerability law: encoding is identical ($A_4 \approx 0$) across families, but both deployment direction and content-steerability sign-flip with the fallback-vs-standing regime; the standing channel (CronusVLA) is steerable whereas the fallback channel (Octo) is inert/content-blind.
Design implication: useful memory augmentation must inject information unique to the past, delivered before the mid-network cutoff $\ell^\star$.

Results¶

$A_1$ (decodable history): 0.389 (Octo-Small), 0.362 (Octo-Base), 0.142 (CronusVLA); $A_4$ (unique history): +0.019, +0.015, −0.011 respectively—$A_4/A_1 \lesssim 0.05$ in all cases.
Shuffled-label probe control returns $A_4 \approx 0$, ruling out overfitting.
Deploy gap $\Delta_\ell$ is significant only through L4 (+0.054, CI [0.028, 0.079] at L4) and collapses at L6 ($\Delta_6 = -0.005$, CI [−0.023, 0.014], n.s.); permutation $p = 0.0004$ at L4, $p = 0.49$ at L6.
Only full occlusion (occ_black) produces a significant deploy gap; partial occlusion, heavy occlusion, and Gaussian blur are all non-significant on the degradation ladder.
Attention-knockout recovers the same L6 cutoff (knockout gap: +0.323 at L0, +0.137 at L4, drops to +0.045 at L6), confirming the localization is method-independent.
Octo-Base deploys history at ≈31% of Octo-Small's normalized rate (inverse scaling), with the same L6 cutoff.
Under the same occlusion, Octo's history contribution rises (fallback) while CronusVLA's falls (standing; normal condition: +0.0498, CI [0.0479, 0.0534]; occlusion: +0.0357, CI [0.0354, 0.0359]—non-overlapping).
CronusVLA injection steer: +0.167 (content-bearing); Octo injection is inert (content-blind) across three experiment variants.
Naturally non-Markov pairs in the frozen stimulus set: only 3/250 cases, confirming the fallback is rarely triggered in distribution.

Limitations¶

$A_4$ residualizes against the raw $t-1$ frame, so abstract history encodings (task progress, intent) that are not a linear read of the previous pixel frame are not captured; the redundancy claim is scoped to the linear-frame hypothesis.
Open-loop evaluation on a small frozen stimulus set (250 pairs, single Bridge dataset) trades behavioral ecological validity for layer-resolved causal localization; episode-level consequences are not measured.
Only two architecture families (Octo/diffusion head; CronusVLA/Qwen backbone) and a two-frame observation window; generalization to longer windows or other VLA families is not established.
Scaling analysis (Octo-Small vs. Octo-Base) is within one architecture family only; whether the fallback-vs-standing regime is settable by design choices beyond architecture is not explored.
Attention-knockout leaves a small but significant residual past L6 under full occlusion (interchange falls to statistical zero there), indicating the two methods are not perfectly aligned post-cutoff.

Relevance to Vision-Language Models¶

This paper provides the first mechanistic account of temporal processing in VLMs extended to action generation, complementing existing work on within-frame vision-language fusion. The key finding—that multi-frame VLMs encode temporal context as a near-redundant copy of the present rather than as independent memory—has direct implications for any VLM architecture that ingests frame sequences (video understanding, multimodal agents, embodied AI). The training-free TDA audit (Algorithm 1) is immediately applicable to other frozen multi-frame VLMs to diagnose whether temporal context is meaningfully utilized or merely parroted. The fallback-vs-standing distinction also raises the question of whether similar deployment-regime differences exist in VLMs used for video QA or temporal reasoning, where the literature similarly assumes that supplying more history is beneficial.