Skip to content

Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory

🕒 Published (v1): 2026-07-20 12:30 UTC · Source: Arxiv · link

Why this paper was selected

Reflective recall cycle for long-term agent memory; novel two-phase (explore + assimilate) design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EAR (Exploratory-Assimilating Reflection) is a framework that makes long-term memory retrieval for LLM agents both high-performing from the start and sample-efficient over time. It combines an UCB-bandit Explorer that performs iterative query-specific search with an Experience Buffer that replays past feedback to update a global residual reranker. On LongMemEval and LoCoMo, it achieves up to 17.9% and 12.5% absolute Recall@5 gains over static retrievers while reaching equivalent performance 77% faster than prior RL-based methods.

Problem

LLM agents are stateless with finite context windows, requiring external memory banks, but existing retrieval methods are brittle: (1) static rerankers trained offline cannot adapt to domain shift or evolving user patterns, and (2) the most recent online RL-based reranker adaptation (Tan et al., 2025) is severely sample-inefficient—it drops ~10% below the static retriever initially and requires hundreds of interactions before recovering. No prior method jointly addresses cold-start performance and long-horizon adaptation efficiency.

Method

EAR learns a residual retrieval adapter \(\phi = \{W_q, W_m\} \in \mathbb{R}^{d \times d}\) applied to frozen encoder embeddings:

\[\tilde{q} = \frac{(I + W_q)q}{\|(I+W_q)q\|_2}, \quad \tilde{m}_i = \frac{(I+W_m)m_i}{\|(I+W_m)m_i\|_2}\]

Exploratory Reflection treats each of the \(K\) retrieved candidates as a bandit arm. Over \(T\) rounds, an Explorer selects a slate of \(s\) memories by maximizing a composite score balancing adapted relevance, slate diversity, and UCB uncertainty:

\[A_i^{(t,j)} = \lambda_{\text{rel}}\,\tilde{q}^\top \tilde{m}_i - \lambda_{\text{div}} D_i^{(t,j)} + \lambda_{\text{exp}} U_i^{(t)}\]

where \(U_i^{(t)}\) is the UCB term and \(D_i^{(t,j)}\) penalizes similarity to already-selected memories. An external critic (LLM citation scoring) returns binary feedback \(y_i \in \{-1, +1\}\).

Assimilating Reflection stores aggregated per-query feedback in an Experience Buffer \(B\) and replays the \(B\) most query-similar historical experiences to update the adapter via a combined loss:

\[\mathcal{L}_{\text{EAR}}^{(n)}(\phi) = \mathcal{L}_{\text{cur}}^{(n)}(\phi) + \lambda_{\text{rep}}\,\mathcal{L}_{\text{rep}}^{(n)}(\phi)\]

Both terms use a feedback-weighted ranking objective. At inference, the Explorer can be disabled (adapter-only) or retained (Explorer-assisted) depending on compute budget.

Key Contributions

  • EAR framework combining UCB-bandit iterative search (Exploratory Reflection) with cross-query experience replay (Assimilating Reflection) for adaptive long-term memory retrieval.
  • 77% reduction in feedback steps to match static retriever performance vs. prior RL-based method (Tan et al., 2025).
  • Demonstrated robustness to noisy LLM feedback (stable even at Precision=70, Recall=70) through multi-trial averaging within Explorer rounds.
  • Heterogeneous memory case study showing EAR adaptively balances episodic (dialogue) and semantic (observational) memory stores, improving downstream QA accuracy by up to 13.6%.
  • Model-agnostic design: compatible with Contriever, Stella, and GTE retrievers, and composable on top of fine-tuned non-RL rerankers.

Results

LongMemEval (Contriever backbone, absolute Recall@5): - EAR: 80.28% vs. Only Retriever: 62.37% (+17.9%) vs. Reranker+RL Update: 59.71% (+20.6%) - LLM-as-Judge: +10.4% over baseline retriever; +15.8% over RL update

LoCoMo (Contriever backbone, absolute Recall@5): - EAR: 55.06% vs. Only Retriever: 42.60% (+12.5%) vs. Reranker+RL Update: 25.97% (+29.1%) - LLM-as-Judge: +9.0% over baseline retriever; +24.4% over RL update

Sample efficiency (LongMemEval, adapter-only inference): - EAR surpasses static retriever after 80 training queries; RL baseline never surpasses it within the 400-query budget - Cold-start degradation: EAR limits to −3% vs. −10% for RL baseline - Under matched LLM-call budget (1600 calls), EAR reaches its peak at ~320 LLM calls vs. ~1400 for the RL baseline

Ablation (LongMemEval / LoCoMo Recall@5): - Adapter only (Tan et al.): 54.12 / 25.97 - + Only Explorer during training: 69.91 / 39.99 - + Only Experience Buffer: 66.32 / 29.54 - EAR w/o Explorer at test time: 72.80 / 43.56 - EAR (full): 80.28 / 55.06

Limitations

  • Adapter parameter count scales as \(2 \times d_{\text{embed}}^2\), meaning larger retrievers (Stella, GTE) require more training queries; EAR with Contriever outperforms EAR with larger retrievers on LongMemEval's smaller training split.
  • Explorer-assisted inference requires \(T-1\) extra LLM critic calls per query (default \(T=4\)), adding latency in production.
  • Evaluation relies primarily on a probabilistic reward simulator (Precision 88.0, Recall 86.0) rather than live LLM calls for systematic noise analysis; real-LLM validation is limited to spot checks.
  • Benchmarks (LongMemEval, LoCoMo) are exclusively chat/dialogue domains; generalization to non-conversational agent memory (e.g., tool-use logs, code history) is unvalidated.
  • A mild cold-start degradation (−3% Recall@5) remains before the adapter warms up.

Relevance to Agentic AI / LLM Agents

Persistent, adaptive long-term memory is a foundational bottleneck for LLM agents operating across extended interactions—EAR directly addresses this by coupling query-time exploration with continual reranker adaptation, moving beyond static RAG toward truly online retrieval. The bandit-based Explorer enables high-quality memory retrieval from the very first interactions (cold-start), while Experience Replay allows the agent to generalize across heterogeneous memory types (episodic vs. semantic), which is critical in real deployments. The 77% sample-efficiency gain over the prior RL approach significantly reduces the warm-up cost that makes online adaptation impractical in short-lived agent sessions. For researchers studying agent memory architectures, this work establishes a strong new baseline and demonstrates that cognitive replay analogies (short-term/long-term memory consolidation) translate into concrete, measurable engineering wins.