Skip to content

MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization

🕒 Published (v1): 2026-03-26 23:28 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; MemoryCD: first benchmark for long-context cross-domain agent memory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MemoryCD is the first large-scale, real-user, cross-domain benchmark for evaluating lifelong user memory in LLM agents, derived from Amazon Review data spanning years of authentic interactions. It evaluates 14 frontier LLMs × 6 memory methods on 4 personalization tasks under single- and cross-domain settings. Current memory mechanisms remain far from satisfying real user preferences, and no single method dominates across tasks.

Problem

Existing memory benchmarks for LLM agents suffer from three compounding failures: (1) most rely on synthetic, LLM-simulated user data rather than authentic behavioral histories; (2) those with real user records (LaMP, MSC) have short context windows (~1K tokens, 1–5 sessions) that do not stress modern million-token LLMs; and (3) evaluation is narrowed to retrieval tasks, ignoring downstream personalization quality and user satisfaction. No prior benchmark evaluates cross-domain memory transfer — a realistic scenario where user preferences expressed in one domain (e.g., Electronics) must inform decisions in another (e.g., Home & Kitchen).

Method

MemoryCD is built from the Amazon Review corpus (Hou et al., 2024). Each user \(u\) is represented as a long-horizon behavioral record pool: $\(M_u = \{(i, m_i, r_{u,i}, t_{u,i}, x_{u,i})\}_{i \in I_u}\)$ where \(r_{u,i} \in \{1,\ldots,5\}\) is the rating, \(t_{u,i}\) the timestamp, and \(x_{u,i}\) the textual review. Users are filtered by domain-specific minimum interaction thresholds (50–1,000 interactions) to ensure rich, long-context memory inputs (~400K tokens, 1K+ sessions per user).

Four personalization tasks formalize the evaluation as \(f_\theta: (M_u, X) \to Y\): 1. Rating prediction — predict \(\hat{r}_{u,i}\); metrics: MAE, RMSE. 2. Item ranking — predict preference ranking \(\pi_u(C)\) over candidates; metric: NDCG@K. 3. Review summarization — generate a short title reflecting user tone; metrics: ROUGE-L, BLEU-1. 4. Review generation — generate full review text matching user style; metrics: ROUGE-L, BLEU-1.

Two memory settings are evaluated: single-domain (memory and target from the same domain) and cross-domain (memory sourced from other domains, testing cold-start transfer). Six memory baselines are implemented: direct long-context prompting, LoCoMo, Mem0, ReadAgent, MemoryBank, and A-Mem, paired with 14 LLMs from GPT-5, Claude-4, and Gemini-2.5 families across 12 domains (main experiments on 4: Personal Care, Books, Electronics, Home & Kitchen).

Key Contributions

  • First benchmark grounded in real, multi-year, cross-domain user interaction histories (vs. synthetic personas in all prior work).
  • End-to-end evaluation pipeline covering all four personalization task types (not just retrieval), with ground truth derived from real user actions.
  • Comprehensive evaluation matrix: 14 LLMs × 6 memory methods × 4 tasks × 12 domains × 2 memory settings.
  • Empirical analysis demonstrating that cross-domain memory transfer efficacy depends on semantic alignment between source and target domains (Electronics → Home & Kitchen transfers well; Books → Home & Kitchen does not).

Results

  • No single memory method dominates across all four tasks in any domain.
  • Long-context prompting is competitive for rating prediction (GPT-5 achieves best MAE of 0.330 in Books with raw prompting vs. all memory methods); memory abstraction methods help most for retrieval-sensitive and generation tasks.
  • Mem0 + Claude-4 Sonnet improves summarization ROUGE-L from 0.145 to 0.202 in Books over raw long-context.
  • A-Mem + Claude-4 Sonnet raises summarization BLEU-1 from 0.089 to 0.150.
  • LoCoMo improves GPT-5 ranking NDCG@1 from 0.146 to 0.232 in Books.
  • Cross-domain: Electronics memory improves GPT-5 MAE on Home & Kitchen from 0.43 (no memory) to 0.30; Claude-4 Sonnet from 0.9 to 0.55.
  • Aggregating all 3 cross-domain sources (Books + Electronics + Personal Care) improves ranking for all backbones on Home & Kitchen vs. no-memory baseline, but can hurt error-based metrics by introducing domain-mismatched noise.
  • Gemini-2.5 Pro leads on generation tasks (e.g., ROUGE-L 0.162 in Books review generation); GPT-5 leads on rating and ranking.

Limitations

  • Evaluation domains in cross-domain experiments are restricted to 4 (Personal Care, Books, Electronics, Home & Kitchen) due to the trade-off between user population size and history length.
  • Metrics (ROUGE-L, BLEU-1) for text generation do not fully capture stylistic or preference alignment; LLM-as-a-judge was deliberately excluded due to scalability, inconsistency, and self-preference bias.
  • Performance levels across all models and methods remain far from human-level user satisfaction on all tasks — the benchmark exposes the gap but does not close it.
  • Cross-domain memory utility is highly domain-pair-dependent; the study is limited to Amazon product reviews and may not generalize to other behavioral modalities (e.g., multimedia, health).
  • Memory methods studied are existing baselines; no new memory architecture is proposed.

Relevance to Agentic AI / LLM Agents

MemoryCD directly addresses the memory subsystem of lifelong LLM agents — a core capability required for agents to act as persistent personal assistants rather than stateless tools. The cross-domain evaluation setting (cold-start personalization from heterogeneous prior interactions) maps precisely onto real-world agentic deployment where a single agent must serve a user across diverse tasks and contexts. The finding that no current memory mechanism solves the problem, and that raw long-context prompting sometimes beats structured memory abstraction, is a critical empirical signal for agent memory architecture research. The benchmark fills a key gap in the agent evaluation ecosystem by providing real user ground truth, enabling principled comparison of memory compression, retrieval, and update strategies in downstream task performance rather than proxy retrieval accuracy.