Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; M3-Agent: multimodal agent with episodic and semantic long-term memory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

M3-Agent is a multimodal agent framework that continuously ingests video/audio streams, builds an entity-centric long-term memory graph (episodic + semantic), and uses RL-trained multi-turn reasoning to answer questions over that memory. A companion benchmark, M3-Bench, evaluates memory-dependent reasoning across robot-perspective and web-sourced long videos. M3-Agent outperforms the strongest prompting-based baseline (Gemini-1.5-pro + GPT-4o hybrid) by 6.7%/7.7%/5.3% on M3-Bench-robot/M3-Bench-web/VideoMME-long.

Problem¶

Existing long-video understanding methods process finite, offline clips and lack persistent world knowledge: they cannot handle arbitrarily long streams online, and they represent entities ambiguously (e.g., "a man with a beard"), causing identity drift and inconsistency across long contexts. Existing LVQA benchmarks test shallow perceptual skills (action recognition, spatial/temporal perception) rather than higher-level memory-dependent reasoning such as person understanding, cross-modal inference, and general knowledge extraction.

Method¶

M3-Agent runs two concurrent pipelines sharing a multimodal long-term memory graph:

Memorization pipeline. Video is processed clip-by-clip (∼30 s each). Perception tools (face detection, speaker diarization) extract face/voice identities and assign persistent IDs via search_node lookups in the memory graph. An MLLM then generates (a) episodic memory: timestamped event descriptions referencing <face_id>/<voice_id> tokens, and (b) semantic memory: abstracted knowledge (preferences, relationships, object rules). Each item is stored as a typed node (text/image/audio) in an entity-centric graph; re-encountered items increment a reliability weight; conflicts are resolved at inference time by weighted voting.

Control pipeline. Given a question \(q\), a policy model \(\pi_\theta\) executes Algorithm 1: a multi-turn loop (max \(H\) rounds) that interleaves [Search] tool calls (returning top-\(k\) nodes or clips from the memory graph) with chain-of-thought reasoning, terminating on [Answer]. The policy is trained with reinforcement learning (reward: correctness of final answer) rather than supervised prompting, enabling adaptive, focused retrieval across rounds.

Memory retrieval supports two granularities: search_node (multimodal query → top-\(k\) nodes) and search_clip (query → top-\(k\) 30 s episodic+semantic clip summaries).

Key Contributions¶

M3-Agent: an online, streaming multimodal agent with a persistent entity-centric memory graph; decouples memorization from control into two concurrent workflows.
M3-Bench: LVQA benchmark with 100 robot-perspective videos (1,276 QA pairs, avg 2,039 s) and 920 web videos (3,214 QA pairs, avg 1,630 s); five question types (multi-evidence reasoning, multi-hop reasoning, cross-modal reasoning, person understanding, general knowledge extraction); all open-ended, manually annotated.
RL-trained control policy enabling multi-turn, iterative memory retrieval superior to single-turn RAG and prompting.
Weighted voting conflict resolution for maintaining consistency in incrementally constructed memory graphs.

Results¶

M3-Bench-robot: M3-Agent +6.7% accuracy vs. Gemini-GPT4o-Hybrid (strongest baseline).
M3-Bench-web: +7.7% accuracy vs. Gemini-GPT4o-Hybrid.
VideoMME-long: +5.3% accuracy vs. Gemini-GPT4o-Hybrid.
Ablation: removing semantic memory causes the largest accuracy drop across all tasks; removing RL (replacing with prompting) degrades multi-turn reasoning performance.
GPT-4o automatic evaluator agrees with human majority vote on M3-Bench at 96% on a 100-sample validation set.

Limitations¶

Memorization relies on external perception tools (face detection, speaker diarization) that may fail in unconstrained environments or with occluded faces/overlapping speech.
The benchmark is bounded: M3-Bench-robot is only 100 videos filmed by actors simulating robots, limiting ecological validity.
Maximum reasoning rounds \(H\) is fixed; very complex queries may exceed the budget without a correct answer.
Training the RL policy requires reward signal (ground-truth answers), which may be expensive to obtain for new domains.
The paper does not report latency or memory-graph scaling behavior at extended operation times (hours/days).

Relevance to Harnesses / Meta-Harnesses¶

M3-Agent is architecturally a domain-specific agent harness: it orchestrates a fixed pipeline of perception sub-tools (face detection, speaker diarization, MLLM), a persistent external state store (the memory graph), and a RL-trained tool-calling loop, all coordinated across two concurrent workflows. The control loop (Algorithm 1) is directly analogous to a harness's dispatch loop — it decides which tool to invoke, collects results, and routes them back into context for the next reasoning step. For researchers tracking harness design, this paper exemplifies how a harness can be trained end-to-end via RL rather than prompt-engineered, and how separating memorization (continuous background ingestion) from control (on-demand query resolution) maps cleanly onto a producer–consumer harness architecture with persistent shared state.