Skip to content

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Julian McAuley (UCSD); incremental multi-turn benchmark for agent memory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MemoryAgentBench is a new benchmark grounded in cognitive science that evaluates LLM-based memory agents across four competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. It transforms existing long-context datasets into incremental multi-turn formats and introduces two new datasets, revealing that no current memory agent achieves strong performance across all four dimensions.

Problem

Existing agent benchmarks over-index on reasoning, planning, and tool use; memory—how agents memorize, update, and retrieve long-term information—is systematically under-evaluated. Prior long-context benchmarks (LongBench, ∞-Bench, NovelQA) deliver the entire context as a single block, which does not reflect the incremental, chunk-by-chunk processing of real memory agents. The closest memory-specific benchmark, LongMemEval, covers only accurate retrieval and uses synthetic conversations with limited topical diversity. No existing benchmark jointly evaluates all four core memory competencies across heterogeneous agent architectures.

Method

The authors ground four competencies in classical memory science (James 1890, McClelland 1995, Anderson & Neely 1996, Wimber 2015): 1. Accurate Retrieval (AR) — one- or multi-hop extraction of the correct passage. 2. Test-Time Learning (TTL) — acquiring new skills (e.g., intent classification, recommendation) from in-context examples without retraining. 3. Long-Range Understanding (LRU) — integrating information spread across \(\geq 100\text{k}\) tokens for global comprehension. 4. Selective Forgetting (SF) — overwriting stale facts with contradictory later facts.

Dataset construction: Existing long-context corpora are segmented into chunks \(c_1, c_2, \ldots, c_n\) and fed to agents sequentially in a simulated user–assistant dialogue with memorization instructions. Two new datasets are introduced: EventQA (automated NIAH-style timeline recall over novels, average 534K tokens) and FactConsolidation (counterfactual edit pairs from MQUAKE ordered so the newer contradicting fact appears later; single-hop FC-SH and multi-hop FC-MH variants at up to 262K tokens). After all chunks are consumed, questions \(q_1, \ldots, q_m\) are posed in batch.

Agent taxonomy evaluated: (1) Long-Context Agents — FIFO context buffer up to window limit; (2) RAG Agents — Simple (BM25/BMX), Embedding-based (Contriever, text-embedding-3, Qwen3-Embedding-4B), Structure-Augmented (RAPTOR, GraphRAG, MemoRAG, HippoRAG-v2, Mem0, Cognee, Zep); (3) Agentic Memory Agents — iterative retrieve-reflect loops (Self-RAG, MemGPT, MIRIX).

Key Contributions

  • MemoryAgentBench: 2 071 questions across 12 dataset splits, covering all four memory competencies and three agent categories (Long-Context, RAG, Agentic Memory).
  • EventQA (new): automated pipeline for temporal-sequence recall over novels; scalable to arbitrary novel-style texts.
  • FactConsolidation (new): SF dataset built from MQUAKE counterfactual pairs, single- and multi-hop variants.
  • Unified evaluation framework: standardized chunk-by-chunk injection protocol with consistent prompt templates across agent types; open-sourced.
  • Empirical study: systematic comparison of 20+ agent implementations revealing competency-specific failure modes.

Results

From Table 3 (overall scores are unweighted averages across all four competency averages):

  • GPT-5-mini (400K long-context): best long-context agent overall — AR 74.4, TTL 48.6, LRU 66.2, SF 53.0, Overall 60.6
  • Claude-3.7-Sonnet (200K): AR 59.7, TTL 53.9, LRU 62.2, SF 22.5, Overall 49.6
  • GPT-4o (128K): AR 58.1, TTL 50.0, LRU 54.9, SF 32.5, Overall 48.8
  • Gemini-2.0-Flash (1M): AR 65.1, TTL 46.4, LRU 41.6, SF 16.5, Overall 42.4
  • BM25 simple RAG: AR avg 60.5 (SH 66, MH 56, LME 45.3, EventQA 74.6), strongest simple baseline for AR
  • HippoRAG-v2 (embedding RAG): highest AR among RAG agents — SH 76, MH 66, LME 50.7
  • MIRIX (agentic, GPT-4o-mini): AR 53.4, compared to MIRIX (4.1-mini): SH 73, MH 75, LME 51.0
  • SF is universally weak: FC-MH scores below 30 for nearly all agents including 1M-context models; GPT-4o scores only 5.0 on FC-MH
  • No agent achieves simultaneously strong performance across all four competencies

Limitations

  • Benchmark focuses exclusively on textual and external-database memory; parametric memory (MemoryLLM, M+, SELF-PARAM) is out of scope.
  • SF evaluation relies on explicit serial-number guardrails in the prompt to signal fact ordering; this may overestimate real-world forgetting ability where such cues are absent.
  • Detective QA (LRU) contains only 71 questions across 10 novels, limiting statistical power.
  • Questions are posed after all chunks are consumed; true online interleaved query–update scenarios (where queries arrive mid-stream) are not tested.
  • The paper text is truncated, so full RAG and agentic agent results on TTL and LRU competencies are not visible in the provided excerpt.

Relevance to Agentic AI / LLM Agents

Memory is the neglected pillar of agentic capability: while planning and tool use benchmarks are mature, MemoryAgentBench provides the first systematic, multi-competency testbed that forces agent architects to confront the distinct failure modes of context-based, RAG-based, and agentic-loop memory strategies. The incremental chunk-injection protocol faithfully models how deployed agents accumulate state over long sessions, distinguishing this from static long-context evaluations. The finding that selective forgetting is universally poor—even in models with 1M-token windows—directly challenges the assumption that large context windows subsume memory; it motivates dedicated update/overwrite mechanisms. For practitioners building persistent agents (assistants, co-pilots, long-running autonomous systems), this benchmark provides actionable signal on which competency their architecture is weakest at.