Skip to content

WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning

🕒 Published (v1): 2025-12-02 05:14 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WorldMM is a multimodal memory agent for long-video QA (hours to weeks) that constructs three complementary memory stores—episodic (multi-scale event KGs), semantic (evolving relational KG), and visual (frame embeddings + timestamp index)—and uses an iterative retrieval agent that adaptively selects the right memory type and temporal granularity per query. It achieves an average 8.4% gain over the prior state-of-the-art across five long-video benchmarks without processing all frames.

Problem

Existing long-video LLMs cannot scale to hour- or week-long videos due to context limits and the loss of visual detail when all frames are uniformly sampled. Memory-augmented retrieval methods mitigate this but rely almost exclusively on textual captions, ignore rich visual evidence, and retrieve a fixed number of clips at a fixed temporal scale—making them brittle when queries demand either fine-grained visual details or varying temporal scopes.

Method

WorldMM operates in three stages:

  1. Multimodal Memory Construction — three parallel memories are built offline from the video stream:
  2. Episodic memory \(\mathcal{M}_e = \{G_{t_0}, \ldots, G_{t_N}\}\): a hierarchy of knowledge graphs (factual entity–action–entity triplets) at increasing temporal granularities (e.g., 30 s, 3 min, 10 min, 1 hr for EgoLifeQA).
  3. Semantic memory \(\mathcal{M}_s\): a single continuously evolving KG that merges relational/habitual triplets via an LLM-driven consolidation process that identifies outdated or conflicting edges and applies targeted updates.

  4. Visual memory \(\mathcal{M}_v = \mathcal{M}_v^f \cup \mathcal{M}_v^I\): dense visual embeddings per fixed-length segment (VLM2Vec-V2 encoder) for keyword-based cosine retrieval, plus a frame-timestamp index for direct temporal access.

  5. Adaptive Memory Retrieval — a retrieval agent \(\mathcal{R}(q, r_{<i})\) iterates, at each step selecting one memory type \(m_i \in \{\mathcal{M}_e, \mathcal{M}_s, \mathcal{M}_v\}\) and a query \(q_i\), or emitting STOP when sufficient information is gathered. Episodic retrieval uses Personalized PageRank (PPR) across all temporal scales followed by an LLM cross-scale reranker to pick the optimal time range. Semantic retrieval uses edge-weighted PPR. Visual retrieval uses either cosine similarity over embeddings or direct timestamp lookup.

  6. Response Generation — a separate response agent receives the full retrieval history (selected memories, queries, retrieved content) and generates the final answer, keeping retrieval and generation cleanly separated.

Key Contributions

  • Three-memory architecture combining episodic (multi-scale KG), semantic (evolving KG with consolidation), and visual (dual-mode: embedding + timestamp) stores.
  • Adaptive retrieval agent that dynamically selects memory type and temporal granularity per query via iterative reasoning, rather than using fixed retrieval schedules.
  • Multi-scale episodic memory spanning seconds to hours, with PPR-based coarse-to-fine retrieval and LLM cross-scale reranking.
  • Semantic memory consolidation that continuously merges new relational knowledge while removing or revising conflicting triplets.
  • Dual-mode visual memory enabling both semantic keyword search (embedding cosine similarity) and precise temporal access (frame-timestamp index).

Results

All numbers are accuracy (%) on multiple-choice QA unless noted:

  • WorldMM-GPT average: 69.5% vs. best baseline M3-Agent at 55.1% — +8.4% absolute across five benchmarks.
  • EgoLifeQA: WorldMM-GPT 65.6% vs. M3-Agent 53.5% (+12.1%).
  • Ego-R1 Bench: 65.3% vs. M3-Agent 52.0% (+13.3%).
  • HippoVlog: 78.3% vs. HippoMM 71.9% (+6.4%).
  • LVBench: 61.9% vs. M3-Agent 49.3% (+12.6%).
  • Video-MME (Long): 76.6% vs. M3-Agent 55.3% (+21.3%).
  • WorldMM-8B (Qwen3-VL-8B backbone) achieves 59.9% average, outperforming all prior memory-based methods including those with GPT backbones.
  • Temporal retrieval (tIoU): WorldMM-GPT 9.57% avg. vs. next best ~3.6% (EgoRAG/Ego-R1) — >2.5× improvement.
  • Ablation: removing visual memory costs −4.2% average; removing semantic memory costs −23% on HabitInsight; replacing multi-scale episodic KG with fixed single timescale costs −6.1%.
  • Multi-turn retrieval: 5 iterations vs. 1 yields +9.3% on EgoLifeQA.

Limitations

  • Memory construction relies on GPT-4o-mini (captioning, KG generation, consolidation), adding offline cost proportional to video length.
  • Inference latency increases with more retrieval iterations; although favorable compared to full-frame methods, exact per-query cost is query-dependent and not fully characterized.
  • Episodic memory retrieval via graph PPR underperforms visual memory in purely perceptual tasks (object details, spatial context), indicating the KG abstraction still loses fine-grained visual information.
  • Indexing visual frames into a structured representation remains harder than textual KGs; visual-only retrieval performs 20% worse than episodic on average.
  • Temporal scales (e.g., 30 s, 3 min, 10 min, 1 hr) are set manually per dataset, requiring dataset-specific tuning.
  • No ablation on audio modality despite HippoVlog containing audio streams.

Relevance to Agentic AI / LLM Agents

WorldMM is a direct instantiation of the agentic memory-augmented reasoning paradigm: an LLM-driven retrieval agent iteratively decides what to query and from which heterogeneous store, mirroring tool-use and function-calling patterns central to agentic systems. The separation of episodic, semantic, and visual memories maps onto classic cognitive memory taxonomies and generalizes beyond video—it demonstrates how agents can maintain and selectively access structured, typed external memories to handle tasks with long temporal horizons. The adaptive STOP mechanism and multi-turn retrieval loop are a concrete example of agentic self-termination and iterative refinement. For researchers tracking LLM agents, WorldMM offers a well-ablated case study of multi-memory heterogeneous retrieval that informs design choices for any agent needing to manage large, multimodal knowledge bases.