Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

🕒 Published (v1): 2026-07-15 08:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MEMCON reformulates LLM agent memory access as a Markov Decision Process and learns an online contextual bandit policy that adaptively decides when and how to retrieve, inject plans, or consolidate memory. It wraps any existing memory backend without modifying it, adds zero extra LLM calls, and consistently outperforms fixed-pipeline memory baselines across 6 benchmarks and 3 agent frameworks.

Problem

Existing LLM agent memory systems use static, hand-coded heuristics: a single global retriever with fixed top-\(k\), fixed graph hop depth, and fixed query templates applied uniformly regardless of task phase, memory size, or agent state. This one-size-fits-all approach is miscalibrated for at least one of four distinct regimes: (i) early tasks where memory is sparse and retrieval dilutes the prompt, (ii) recurring goals where plan reuse beats nearest-neighbor retrieval, (iii) stuck agents where re-retrieval with an alternative query is needed, and (iv) long task streams where the memory store must be consolidated and pruned. Adaptive alternatives (e.g., MemGPT) exist but require an additional LLM call per memory operation.

Method

MEMCON defines a Memory MDP \(\mathcal{M}_{\text{mem}} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)\) where the state \(s = (s_{\text{task}}, s_{\text{mem}})\) captures task progress signals (goal type, step phase, is-stuck flag, objects held, locations visited) and memory status (memory size, plan availability, learning phase). The action space is:

\[\text{op} \in \{\texttt{RETRIEVE}, \texttt{PLAN\_INJECT}, \texttt{RE-RETRIEVE}, \texttt{CONSOLIDATE}, \texttt{FORGET}, \texttt{NOOP}\}\]

with parameters \(\theta = (\text{top-}k, \text{insight-}k, \text{hop})\). The state is discretized to a hashable key \(\phi(s)\) yielding \(O(\text{hundreds})\) distinct states per benchmark.

The online policy uses UCB action selection:

\[a_t = \arg\max_{a \in \mathcal{A}} \left[ Q(\phi(s_t), a) + c\sqrt{\frac{\ln N(\phi(s_t))}{N_a(\phi(s_t))}} \right]\]

Q-values are warm-started from interpretable priors (RETRIEVE and PLAN_INJECT positive, FORGET and NOOP mildly negative) and updated after each episode via reverse-discounted Monte Carlo:

\[Q(\phi_j, a_j) \leftarrow Q(\phi_j, a_j) + \alpha\left[\gamma^{|ep|-j-1} \cdot r_i - Q(\phi_j, a_j)\right]\]

Episode reward includes a success/failure term plus an efficiency bonus: \(r(\tau_i) = r_{\text{succ}} \cdot \mathbb{1}[\text{success}] + \lambda \cdot \max(0, 1 - T_i/T_{\text{max}}) - r_{\text{fail}} \cdot \mathbb{1}[\text{failure}]\).

MEMCON is implemented as a backend-agnostic thin wrapper intercepting two abstract entry points (retrieve, store) of any memory backend. Two augmented operations are added: generalized plan injection (regex-rewrite instance-specific identifiers from successful trajectories into reusable templates stored in a JSON index) and goal decomposition (a deterministic heuristic splitting composite multi-object tasks). The Q-table is persisted to disk between tasks; no GPU or pretraining is required.

Key Contributions

• Formalizes agent memory management as a Memory MDP, cleanly separating the control policy (how to access) from the storage backend (what is stored).
• Introduces a lightweight tabular contextual bandit with UCB exploration, warm-start priors, and reverse-discounted credit assignment that converges within tens of tasks using zero extra LLM calls.
• Provides a backend-agnostic wrapper compatible with vector stores, skill libraries, graph memories, latent-token memories, and summarization-based memories unchanged.
• Introduces PLAN_INJECT and RE-RETRIEVE as learnable MDP actions targeting plan reuse and stuck-agent recovery.
• Comprehensive evaluation: 6 benchmarks × 3 agent frameworks × 3 LLM backbones × 9 memory baselines.

Results

• ALFWorld (Lobster, GPT-4.1-mini): MEMCON 67.9% vs. best baseline G-Memory/ExpBank 59.7% (+8.2 pp); no-memory baseline 43.3%.
• ALFWorld (Agent-FW, GPT-4.1-mini): MEMCON 71.0% vs. best baseline G-Memory 70.2%.
• PDDL (LangGraph): MEMCON 40.0% vs. best baseline Generative/ExpBank 35.0%.
• GAIA (Agent-FW): MEMCON 23.0% vs. best baseline G-Memory 20.0%.
• Average across 6 benchmarks: MEMCON 42.6–43.6% (per framework) vs. best baseline 38.2–40.8%; gains up to 15.2 points over best fixed-pipeline baselines.
• Token consumption reduced 5–20% relative to baselines despite equal or higher task success (gains are not attributable to retrieving more).
• Results consistent across GPT-4.1-mini, Claude Sonnet-4, and DeepSeek-V3.2 backbones (per-backbone tables in Appendix B).
• Component ablation confirms the learned controller alone accounts for the majority of gains; plan injection and goal decomposition are additive but optional.

Limitations

• Tabular discretization of the state space requires manual bin/threshold design per benchmark (exact values in Appendix C); generalization to out-of-distribution goal types may require re-binning.
• The contextual bandit treats the MDP as episode-level feedback (Monte Carlo, no within-episode bootstrapping), which may slow adaptation when episodes are long and the terminal reward is weakly correlated with individual memory decisions.
• PLAN_INJECT's regex-based identifier generalization degrades to identity on novel vocabularies; no learned abstraction is used.
• Evaluation covers six benchmarks but all are either household/interactive (ALFWorld, PDDL, ScienceWorld) or short-context QA/web tasks; long-horizon, open-ended agentic tasks (e.g., multi-day research agents) are not assessed.
• The wrapper requires backends to expose a standardized two-method interface; backends with non-standard APIs need adaptation shims.
• Per-backbone numerical results for Sonnet-4 and DeepSeek-V3.2 are deferred to the appendix and not reproduced in the main text.

Relevance to Harnesses / Meta-Harnesses

MEMCON is directly relevant to harness design because it treats memory management as a learnable control layer that sits above any agent backend — exactly the abstraction level a meta-harness operates at. By casting memory decisions (retrieve, inject plan, consolidate) as a contextual bandit, it gives harness builders a principled, trainable alternative to the hard-coded memory pipelines most frameworks ship today, without touching the underlying LLM or memory store. The zero-extra-LLM-calls constraint is practically significant: it means the policy adds negligible latency and cost overhead, making it deployable inside a harness orchestration loop rather than only in offline settings. For someone building agent harnesses, the key takeaway is that memory-access scheduling is a first-class control problem that can be learned from task feedback, and MEMCON provides both the formalism and a working implementation tested across three agent frameworks.

🔗 Related in this collection

• G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems