Skip to content

Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Memento: proactive assistant for ultra-long streaming video

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Memento is the first proactive vision-language framework for ultra-long (up to 7-hour) streaming video that escapes the token-accumulation trap of prior online video LLMs via a dynamic, bounded memory system. It introduces Dynamic Memory with a Remember-and-Forget gate, query-conditioned retrieval, and a step-aware training mask that aligns supervision to dynamically evolving memory states. A new 54K-sample dataset and benchmark (MementoBench) support training and evaluation of long-range proactive interaction.

Problem

Existing online video LLMs accumulate one set of visual tokens per frame, causing GPU memory to grow linearly with video length and forcing OOM failures after ~25 minutes—far short of the hours-long context needed for an all-day proactive assistant. Fixed-memory models cap retention and cannot proactively respond without an explicit user query. No existing dataset provides long-horizon, timestamped proactive QA supervision beyond a few minutes.

Method

Architecture. Each incoming frame \(f_t\) is encoded by a SigLIP-ViT-L/384 encoder into \(v_t \in \mathbb{R}^{(1+h_p \times w_p) \times C}\). Instead of appending \(v_t\) as new tokens, Memento routes it through:

  1. Dynamic Memory (DM) — Remember-and-Forget gate. Two relevance scores are computed:
  2. Short-term \(\delta\): cosine similarity between \(v_t\) and the most recent memory token \(m_{t-1}\).
  3. Long-term \(\sigma\): cross-attention scalar between \(v_t\) and all historical memory slots \(M_{t-1}\), passed through a sigmoid: \(\sigma = \psi\!\left(\text{softmax}\!\left(\tfrac{v_t W_q (M_{t-1}W_k)^\top}{\sqrt{d}}\right) M_{t-1}W_v W_o\right)\).

  4. If \(\delta > \epsilon\): fuse \(v_t\) into the last memory slot (locally redundant). If \(\delta \le \epsilon\) and \(\sigma > \epsilon\): soft-update across all memory slots (long-term repetition). If both \(\le \epsilon\): append \(v_t\) as a new slot (novel content). Threshold \(\epsilon = 0.7\).

  5. Query-related Memory Selection (QMS). User query tokens \(Q\) cross-attend to all memory slots \(M_t\), producing per-slot relevance scores \(R \in \mathbb{R}^{N_t}\). Top-\(k\) slots (\(k = r_\text{qms} \cdot N_t\), \(r_\text{qms}=50\%\)) form the compact \(M'_t\) fed to the LLM.

  6. Step-Aware Memory Attention (SAMA). During training, a binary mask \(A \in \{0,1\}^{L \times L}\) restricts each token to attend only to memory slots present at its video step, preventing access to future or expired memory—a structural mismatch that makes naive causal attention invalid for dynamic memory sequences.

Training objective combines a language-model loss \(-\log P^{[\text{Txt}]}\) and a streaming loss \(-\log P^{[\text{EOS}]}\) from VideoLLM-online, now applicable thanks to SAMA.

Model: LLaMA-3.1-8B-Instruct backbone, LoRA rank 128, trained 1 epoch on 4 Ă— A100 80 GB.

Key Contributions

  • First proactive VLM framework operating over ultra-long (hours-scale) streaming video without token accumulation.
  • Dynamic Memory with a three-way Remember-and-Forget gate combining short-term cosine similarity and long-term cross-attention gating.
  • Query-related Memory Selection (QMS) for query-aware top-\(k\) slot retrieval, decoupling memory size from LLM input length.
  • Step-Aware Memory Attention (SAMA): a training mask that aligns supervision to temporally valid memory, enabling stable learning for dynamic memory models.
  • Memento-54K: 53.6K streaming QA samples from 4,466 Ego4D videos (5 min–7 h), covering 9 task types (temporal counting, ordering, duration, changing, disappear; spatial counting, abnormal, disappear, appear) across action, object, and text modalities.
  • MementoBench: evaluation protocol with three metrics—TimeRecall (within-5s response rate), Score (GPT-3.5 quality), and Redundancy (spurious response fraction).

Results

  • GPU memory: VideoLLM-online hits OOM (~80.5 GB peak) at ~25 minutes; Memento stays bounded at ≤45.3 GB across 4-hour streams, enabling uninterrupted proactive interaction.
  • MementoBench: Memento outperforms VideoLLM-online* (same dataset, same schedule) across TimeRecall, Score, and Redundancy metrics (exact numbers are cut off in the provided text; full tables are in Appendix/Section 5.2).
  • VideoLLM-online (without Memento-54K data) produces no valid responses beyond ~25 minutes; Memento correctly tracks repeated events (e.g., medication injections) across 1h 40m+ spans.

Limitations

  • Exact quantitative improvement margins over baselines are not fully reported in the provided text excerpt.
  • Memory threshold \(\epsilon\) and top-\(k\) ratio \(r_\text{qms}\) are fixed hyperparameters; sensitivity analysis is deferred to appendix.
  • Dataset built on Ego4D (egocentric, daily-life), which may not generalize to other streaming video domains (surveillance, clinical, driving).
  • Evaluated only against VideoLLM-online as a direct baseline; comparisons to GPT-4o and Gemini 1.5 Pro appear qualitative only.
  • Requires 4 Ă— A100 80 GB GPUs; deployment cost at inference scale for all-day operation is unaddressed.

Relevance to Vision-Language Models

Memento directly extends the frontier of online VLMs by replacing frame-wise token accumulation—the dominant paradigm in VideoLLM-online, VideoLLM-MoD, and LION-FS—with a gated dynamic memory that scales sub-linearly in memory with video length, a fundamental architectural advance for streaming VLMs. The SAMA training technique is broadly applicable to any memory-augmented VLM that evolves representations across time steps, addressing a previously unsolved supervision-alignment problem. The proactive interaction objective (Streaming-EOS) and the new MementoBench evaluation framework push the community toward measuring temporal reasoning and autonomous response triggering, rather than offline QA accuracy, which is a meaningful shift in how VLM capabilities are benchmarked for real-world deployment.