Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents

🕒 Published (v1): 2026-06-25 18:50 UTC · Source: Arxiv · link

Why this paper was selected

Memory-update gap benchmark+training; directly addresses multi-session fact-staleness in deployed agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM agents operating over long multi-session conversations systematically fail to supersede stale facts with updated ones when forced to maintain a bounded, self-written memory. This "supersession gap" is statistically significant, survives frontier-scale models and larger memories, but can be partially closed by GRPO reinforcement-learning fine-tuning on a new open environment called Supersede.

Problem

Long-horizon agents must continuously update their memory as facts change across sessions, but existing work only measures this failure; no trainable environment targets temporal fact-currency as the reward. On the LongMemEval knowledge-update subset, replacing full context with a bounded self-maintained memory (B = 300 chars) drops accuracy significantly even for frontier models. Critically, neither scaling the model nor scaling the memory budget closes the gap, ruling out the two obvious engineering escapes.

Method

Diagnosis. The authors compare two conditions on 78 real LongMemEval knowledge-update questions: (1) F ULL - CONTEXT (all sessions in one context window) and (2) B OUNDED - MEMORY (agent rewrites a notes field of ≤ B characters after each session; raw sessions are never re-fed). They sweep three GPT models and vary history length (∼2 vs. ∼48 sessions) and memory budget (fixed vs. proportional) to isolate the bottleneck.

Environment. Supersede is implemented as a verifiers MultiTurnEnv. The reward is: $\(r = r_{\text{cur}} + \lambda\, r_{\text{stale}}\)$ where \(r_{\text{cur}} = \mathbf{1}[\text{answer conveys current value}]\) (programmatic normalized variant match + token-overlap fallback) and \(r_{\text{stale}} = -\mathbf{1}[\text{answer asserts superseded value}]\). No judge model is needed for the primary reward; \(\lambda = 0\) in the reported runs.

Training. GRPO fine-tuning of Qwen2.5-3B-Instruct (LoRA rank 32, \(\alpha = 64\), lr \(10^{-5}\), batch 32, group 8) on 2,000 procedurally generated 6–8-session synthetic episodes with buried supersession events. Transfer is evaluated on the real held-out LongMemEval oracle split (never seen in training) using the programmatic matcher.

Key Contributions

• Isolation result: bounded memory vs. full context paired McNemar test on real conversational data; identifies memory maintenance (not reading comprehension) as the bottleneck.
• Frontier confirmation: gap persists on gpt-5.4 (92% → 77%, \(p = 0.0033\)); full-context accuracy saturates while bounded-memory lags.
• Scale ≠ size: 24× longer history collapses accuracy 68% → 28%; 24× larger memory (proportional budget ~7,150 chars) recovers none of it (28% → 28%, McNemar \(b = c = 4\)).
• Open RL environment: Supersede, built on verifiers/prime-rl, with a verifiable programmatic supersession reward; first environment whose reward directly targets temporal fact-currency.
• Trainability: GRPO on Qwen2.5-3B nearly doubles held-out supersession accuracy (9.0% → 16.7%, +86% relative), monotonically across checkpoints.

Results

• gpt-4.1-mini: full-context 82% → bounded-memory 63% (\(p = 0.0035\), 19 vs. 4 flips).
• gpt-4.1: 91% → 64%.
• gpt-5.4: 92% → 77% (\(p = 0.0033\), 13 vs. 1 flips).
• Scale study (gpt-4.1-mini, \(n = 25\)): oracle (2 sessions, 300 char) 68% → \(s\) split (48 sessions, 300 char) 28%; \(s\) split with proportional memory (~7,150 char) still 28%.
• GRPO training (Qwen2.5-3B): procedural reward rises 0.66 → 0.97; held-out real-data accuracy 9.0% → 16.7% (+7.7 pts), monotonically from step 150 onward; self-terminates at step 175 on zero-advantage.
• Environment validation: intrinsic programmatic reward on gpt-4.1-mini yields 57.7% vs. 63% by LLM judge (matcher is stricter but consistent).

Limitations

• Single small training model (Qwen2.5-3B, single seed); absolute trained accuracy (16.7%) remains far below frontier full-context ceiling.
• Procedural curriculum saturates before training ends; harder, more implicit episodes needed to extend the curve.
• Scale study uses \(n = 25\) (vs. \(n = 78\) for the main gap); constant-ratio null is noisier.
• "Scale" axis grows surrounding distractors, not necessarily the number of fact updates; these are not equivalent.
• Grading heterogeneity: Section 5.1 uses LLM judge, Sections 5.2–5.3 use programmatic matcher; absolute numbers across sections are not directly comparable.
• Multi-seed statistical significance for the training gain is ongoing work.

Relevance to Agentic AI / LLM Agents

Supersession is a fundamental requirement for any agent operating over persistent, evolving user state — yet this paper is the first to provide a trainable environment with a verifiable reward directly targeting temporal fact-currency, rather than proxies like task success or evidence relevance. The finding that neither model scale nor memory budget closes the gap reframes supersession as a distinct policy skill that must be optimized for, not a capability that falls out of general scaling. The released environment (compatible with the emerging OpenEnv standard) and the GRPO training protocol give the community a concrete mechanism to make agents' self-maintained memories stay current — a critical gap given that all major lab memory systems (OpenAI "Dreaming," Claude memory, Gemini personalization) self-report accuracy well below ceiling on update tasks.