Skip to content

AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Interactive benchmark for memory management in long-horizon assistant conversations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AMemGym is an interactive benchmark for evaluating and optimizing LLM assistant memory in long-horizon conversations. It replaces static, off-policy evaluation datasets with on-policy interactions driven by LLM-simulated users grounded in structured state evolution trajectories. The framework enables both reliable memory assessment and agent self-improvement via environmental feedback.

Problem

Existing memory benchmarks (MSC, LongMemEval, PersonaMem, etc.) use static, off-policy data: the assistant is tested on conversation traces it did not generate itself. This introduces reuse bias—agents' memory operations are tightly coupled to their own conversational choices, so off-policy traces misrepresent true performance and can misdirect optimization (e.g., configuration rankings for hyperparameters like \(n_s\) and \(\text{topk}\) differ substantially between on- and off-policy evaluation). Manual curation also limits scale and diversity.

Method

AMemGym grounds free-form LLM interactions in structured state evolution data generated via a reverse-engineering pipeline:

  1. User profile sampling: 100K personas from NemotronPersonas provide diverse contextual backbones.
  2. Question & schema sampling: For each profile, evaluation questions \(Q_p\) are drawn; an LLM extracts required information types and merges them into a canonical state schema \(\Sigma = \{(s_j, V_j)\}_{j=1}^M\) of \(M\) discrete state variables.
  3. State evolution trajectory: The user's states evolve over \(N_p\) periods via narrative life events \(e_t\), yielding \(T_\sigma = (\sigma_0, \ldots, \sigma_{N_p})\) as ground truth.
  4. Grounded utterances: Pre-generated natural-language utterances \(u_{t,k} = G_{\text{utt}}(\sigma_{\text{exposed}}, p)\) expose state subsets during interaction.
  5. On-policy simulation: A user LLM role-plays based on profile + current state, producing dialogue history \(\tau\); the assistant updates its memory \(m_t\) after each turn.
  6. Periodic evaluation: The assistant answers all \(Q_p\) questions after each period; responses are scored against ground-truth answers \(r_{i,\nu}\) verified via a round-trip LLM classifier.

Metrics include: overall QA accuracy; normalized memory score \(S_{\text{memory}} = \frac{S_{\text{overall}} - S_{\text{random}}}{S_{\text{UB}} - S_{\text{random}}}\); and diagnostic failure rates decomposed into write, read, and utilization failures.

Four memory architectures are compared: Native LLM (in-context raw history), Standard RAG, Agentic Write External (AWE, LLM-extracted summaries + embedding retrieval), and Agentic Write In-Context (AWI).

Key Contributions

  • AMemGym framework: first on-policy, interactive benchmark for conversational assistant memory with fully automated, scalable data generation.
  • Empirical demonstration of reuse bias: direct on-policy vs. off-policy comparison showing rank reversals for memory agent configurations (e.g., AWE-(2,4,30) drops 3 ranks under off-policy; gemini-2.5-flash-lite drops 2 ranks).
  • Comprehensive diagnostic metrics: decompose end-to-end failures into write/read/utilization stages for systematic attribution.
  • Proof-of-concept for agent self-evolution: an agent uses AMemGym's periodic evaluation feedback to autonomously refine its memory management policy.
  • Meta-evaluation: human annotation confirms 99.1% state exposure quality, 99.2% conversational state integrity, and LLM-human judgment agreement of 0.94–0.96.

Results

  • Native LLMs: All models achieve \(S_{\text{UB}} > 0.8\) (strong short-context reasoning), but scores drop sharply as interaction history grows; many fall below 50% of upper bound by later periods; some reach near-random performance. Top model: gemini-3-pro-preview (mean memory score 0.463, overall 0.556).
  • On- vs. off-policy gap: AWE-(2,4,30) scores 0.291 on-policy vs. 0.253 off-policy (rank drops 3); AWI scores 0.172 on-policy vs. 0.199 off-policy (rank rises 1); configuration ranking for \(n_s\) and \(\text{topk}\) reverses between evaluation modes.
  • Agentic memory vs. baselines: AWE variants outperform both native LLMs and standard RAG; AWE-(2,4,30) achieves the highest on-policy memory score (0.291) among memory agents using gpt-4.1-mini.
  • LLM comparison (on-policy): claude-sonnet-4 (0.337 mean memory score) > gemini-2.5-flash (0.327) > gemini-2.5-flash-lite (0.271) > gemini-2.0-flash/gpt-4.1 (0.245/0.240) > deepseek-v3 (0.139).

Limitations

  • LLM-simulated users may not fully replicate real human conversational diversity despite high meta-evaluation scores.
  • Evaluation scope: results reported primarily on the base configuration (10 periods, 2 required states/question, 4 turns/state exposure, 128K+ context); the extra configuration (512K+) results are relegated to appendix.
  • Fixed user LLM (gpt-4.1) for simulation—though a separate study shows minimal sensitivity, this is still a potential confound.
  • Cost: structured data generation and on-policy simulation with GPT-4.1 are non-trivial; the framework's scalability advantage over manual curation depends on LLM API costs.
  • State schema is discrete: continuous or hierarchical personal states may not be well-captured by the discrete variable schema \(\Sigma\).

Relevance to Agentic AI / LLM Agents

AMemGym directly addresses a core infrastructure gap for agentic systems: reliable evaluation of long-horizon memory, which is foundational to personalized assistants, multi-session agents, and any system that must maintain user state across extended interactions. The demonstration that off-policy benchmarks produce systematically misleading rankings for agentic memory configurations is a critical methodological warning for the field—results from static benchmarks cannot be trusted to guide memory architecture decisions. The self-evolution proof-of-concept, where an agent uses the framework's diagnostic feedback to improve its own memory policy, points toward a broader paradigm of environment-driven agent optimization relevant to training and fine-tuning pipelines for LLM agents. The decomposition into write/read/utilization failure modes provides actionable diagnostics for researchers building and debugging memory-augmented agents.