Skip to content

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

🕒 Published (v1): 2026-07-15 08:32 UTC · Source: Arxiv · link

Why this paper was selected

Learned adaptive memory access (not fixed retrieval); novel control-process framing for agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MEMCON reformulates LLM agent memory access as a Markov Decision Process and learns an online contextual bandit policy that adaptively decides when and how to retrieve, inject plans, or consolidate memory. It wraps any existing memory backend without modifying it, adds zero extra LLM calls, and consistently outperforms fixed-pipeline memory baselines across 6 benchmarks and 3 agent frameworks.

Problem

Existing LLM agent memory systems use static, hand-coded heuristics: a single global retriever with fixed top-\(k\), fixed graph hop depth, and fixed query templates applied uniformly regardless of task phase, memory size, or agent state. This one-size-fits-all approach is miscalibrated for at least one of four distinct regimes: (i) early tasks where memory is sparse and retrieval dilutes the prompt, (ii) recurring goals where plan reuse beats nearest-neighbor retrieval, (iii) stuck agents where re-retrieval with an alternative query is needed, and (iv) long task streams where the memory store must be consolidated and pruned. Adaptive alternatives (e.g., MemGPT) exist but require an additional LLM call per memory operation.

Method

MEMCON defines a Memory MDP \(\mathcal{M}_{\text{mem}} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)\) where the state \(s = (s_{\text{task}}, s_{\text{mem}})\) captures task progress signals (goal type, step phase, is-stuck flag, objects held, locations visited) and memory status (memory size, plan availability, learning phase). The action space is:

\[\text{op} \in \{\texttt{RETRIEVE}, \texttt{PLAN\_INJECT}, \texttt{RE-RETRIEVE}, \texttt{CONSOLIDATE}, \texttt{FORGET}, \texttt{NOOP}\}\]

with parameters \(\theta = (\text{top-}k, \text{insight-}k, \text{hop})\). The state is discretized to a hashable key \(\phi(s)\) yielding \(O(\text{hundreds})\) distinct states per benchmark.

The online policy uses UCB action selection:

\[a_t = \arg\max_{a \in \mathcal{A}} \left[ Q(\phi(s_t), a) + c\sqrt{\frac{\ln N(\phi(s_t))}{N_a(\phi(s_t))}} \right]\]

Q-values are warm-started from interpretable priors (RETRIEVE and PLAN_INJECT positive, FORGET and NOOP mildly negative) and updated after each episode via reverse-discounted Monte Carlo:

\[Q(\phi_j, a_j) \leftarrow Q(\phi_j, a_j) + \alpha\left[\gamma^{|ep|-j-1} \cdot r_i - Q(\phi_j, a_j)\right]\]

Episode reward includes a success/failure term plus an efficiency bonus: \(r(\tau_i) = r_{\text{succ}} \cdot \mathbb{1}[\text{success}] + \lambda \cdot \max(0, 1 - T_i/T_{\text{max}}) - r_{\text{fail}} \cdot \mathbb{1}[\text{failure}]\).

MEMCON is implemented as a backend-agnostic thin wrapper intercepting two abstract entry points (retrieve, store) of any memory backend. Two augmented operations are added: generalized plan injection (regex-rewrite instance-specific identifiers from successful trajectories into reusable templates stored in a JSON index) and goal decomposition (a deterministic heuristic splitting composite multi-object tasks). The Q-table is persisted to disk between tasks; no GPU or pretraining is required.

Key Contributions

  • Formalizes agent memory management as a Memory MDP, cleanly separating the control policy (how to access) from the storage backend (what is stored).
  • Introduces a lightweight tabular contextual bandit with UCB exploration, warm-start priors, and reverse-discounted credit assignment that converges within tens of tasks using zero extra LLM calls.
  • Provides a backend-agnostic wrapper compatible with vector stores, skill libraries, graph memories, latent-token memories, and summarization-based memories unchanged.
  • Introduces PLAN_INJECT and RE-RETRIEVE as learnable MDP actions targeting plan reuse and stuck-agent recovery.
  • Comprehensive evaluation: 6 benchmarks × 3 agent frameworks × 3 LLM backbones × 9 memory baselines.

Results

  • ALFWorld (Lobster, GPT-4.1-mini): MEMCON 67.9% vs. best baseline G-Memory/ExpBank 59.7% (+8.2 pp); no-memory baseline 43.3%.
  • ALFWorld (Agent-FW, GPT-4.1-mini): MEMCON 71.0% vs. best baseline G-Memory 70.2%.
  • PDDL (LangGraph): MEMCON 40.0% vs. best baseline Generative/ExpBank 35.0%.
  • GAIA (Agent-FW): MEMCON 23.0% vs. best baseline G-Memory 20.0%.
  • Average across 6 benchmarks: MEMCON 42.6–43.6% (per framework) vs. best baseline 38.2–40.8%; gains up to 15.2 points over best fixed-pipeline baselines.
  • Token consumption reduced 5–20% relative to baselines despite equal or higher task success (gains are not attributable to retrieving more).
  • Results consistent across GPT-4.1-mini, Claude Sonnet-4, and DeepSeek-V3.2 backbones (per-backbone tables in Appendix B).
  • Component ablation confirms the learned controller alone accounts for the majority of gains; plan injection and goal decomposition are additive but optional.

Limitations

  • Tabular discretization of the state space requires manual bin/threshold design per benchmark (exact values in Appendix C); generalization to out-of-distribution goal types may require re-binning.
  • The contextual bandit treats the MDP as episode-level feedback (Monte Carlo, no within-episode bootstrapping), which may slow adaptation when episodes are long and the terminal reward is weakly correlated with individual memory decisions.
  • PLAN_INJECT's regex-based identifier generalization degrades to identity on novel vocabularies; no learned abstraction is used.
  • Evaluation covers six benchmarks but all are either household/interactive (ALFWorld, PDDL, ScienceWorld) or short-context QA/web tasks; long-horizon, open-ended agentic tasks (e.g., multi-day research agents) are not assessed.
  • The wrapper requires backends to expose a standardized two-method interface; backends with non-standard APIs need adaptation shims.
  • Per-backbone numerical results for Sonnet-4 and DeepSeek-V3.2 are deferred to the appendix and not reproduced in the main text.

Relevance to Agentic AI / LLM Agents

MEMCON directly addresses a core bottleneck in long-running LLM agents: the assumption that a fixed retrieval heuristic is appropriate across all task phases and memory states. By casting memory access as a learned MDP policy, it provides a principled, lightweight mechanism for agents to self-regulate their own context consumption, which is directly relevant to improving both capability and cost-efficiency in production agentic systems. The backend-agnostic design means the control layer can be dropped onto any existing memory implementation (Voyager skill libraries, G-Memory graphs, MemGPT-style stores) without architectural changes, lowering adoption friction. The demonstration that a tabular bandit with tens of tasks of online learning outperforms all static baselines—including the strongest graph-based memory (G-Memory)—suggests that adaptive control is a higher-leverage intervention than improvements to memory representation alone, which has implications for how the community prioritizes future memory research.