Skip to content

What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

🕒 Published (v1): 2026-07-09 01:15 UTC · Source: Arxiv · link

Why this paper was selected

Rate-distortion theory for agent memory compaction; principled framework for KV/context management

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This survey unifies four previously siloed memory-compaction fields (KV-cache compression, prompt compression, architectural bounded memory, and agent memory) under a single rate–distortion objective, derives a layer-agnostic lower bound on compaction error, and proposes a cross-layer benchmark (COMPACT-Bench) that measures error accumulation under repeated compaction—the regime agents actually operate in.

Problem

Four research communities (KV-cache compression, prompt/context compression, bounded-memory architectures, agent memory consolidation) solve the same problem independently, with incompatible metrics, no cross-layer mechanism transfer, and no benchmark covering repeated compaction as agents perform it. The field lacks a shared theoretical yardstick, a unified failure-mode analysis, and any measurement of how iterative compaction compounds errors over time.

Method

The paper formalizes memory compaction as a single rate–distortion optimization: given history \(H\), produce compact representation \(Z\) with \(\text{rate}(Z) \leq B\) to minimize expected task loss:

\[\min_\theta \; \mathbb{E}_{(H,Q,Y)}\left[\ell\!\left(U(C_\theta(H), Q),\, Y\right)\right] \quad \text{s.t.} \quad \text{rate}(Z) \leq B.\]

A layer-agnostic lower bound on error probability is derived via the data-processing inequality and Fano's inequality:

\[P_e \geq \frac{H(Y|Q) - B - 1}{\log |\mathcal{Y}|}, \quad \text{whenever } B < I^\star(Q),\]

where \(I^\star(Q) = I(Y; H | Q)\) is the task-conditioned information requirement. From this bound the paper extracts three first-class method properties—reversibility (P-rev), query-conditioning (P-q), and fidelity profile (P-fid)—and a seven-axis taxonomy (granularity, lifecycle stage, lossiness, adaptivity, learnability, mechanism, storage substrate) that classifies ~70 methods uniformly across all four layers. Concrete mechanisms are then ported cross-layer: e.g., Quest's no-eviction retrieval pattern applied to agent memory design; Ada-KV's output-error bound applied as a summarization stopping rule. COMPACT-Bench is proposed and partially run to place KV eviction, quantization, prompt compression, and summarization on one bytes-per-token axis while measuring error accumulation under repeated compaction on commodity hardware.

Key Contributions

  • Formal unification of all four compaction layers as one rate–distortion problem with a provable, layer-agnostic lower bound (Eq. 2) that quantifies the penalty of query-agnostic compaction as the query entropy \(H(Q)\).
  • Seven-axis taxonomy classifying ~70 methods uniformly across layers, with per-method tables in an appendix.
  • Explicit cross-layer mechanism transfer: five compaction-aware design principles derived by porting attention-score eviction logic, retrieval-backed reversibility, and output-error bounds between the serving stack and agent memory.
  • Four falsifiable predictions (one per layer) derived from the bound, two of which (KV needle-recall collapse, agent error super-linear growth under repeated summarization) are tested in a reference experiment.
  • COMPACT-Bench proposal: a unified benchmark placing KV eviction, quantization, prompt compression, and summarization on one budget axis with repeated-compaction measurement—neither property exists in any current benchmark.
  • Consolidated open-problem agenda across all layers.

Results

  • Reference experiment (run on commodity hardware) confirms prediction (1): attention-score KV eviction causes needle recall to collapse once per-token budget falls below the needle's share of attention mass.
  • Reference experiment confirms prediction (4): under repeated irreversible summarization, agent end-task error grows super-linearly in the number of compaction events, while retrieval-backed reversible memory stays flat.
  • No new SOTA numbers are reported; the paper is a survey/framework/benchmark-proposal contribution. Prior-work numbers are cited as provisional where unrefereed.

Limitations

  • COMPACT-Bench reference experiments are small-scale and run on commodity hardware; results are illustrative, not a full benchmark.
  • Survey scope is restricted to context-level compaction (2023–2026); weight-level compression (pruning, distillation, weight quantization) and long-context training without memory bounds are excluded.
  • Many cited preprint numbers are flagged as provisional and unreproduce; no independent replication is performed.
  • The unified lower bound assumes query-agnostic compaction is Markov (\(Y - H - Z\) given \(Q\)); query-conditioned methods require separate analysis and the bound's tightness in practice is not characterized.
  • Cross-layer mechanism transfer is argued theoretically and by analogy; no ablation confirms that porting, e.g., Quest's retrieval pattern to agent memory actually yields gains.

Relevance to Agentic AI / LLM Agents

This survey is directly relevant because agents are the tier where the formalism's core failure mode—irreversible, query-agnostic compaction applied repeatedly—is most damaging: each summarization or context-curation step discards information that later turns may need, and errors compound super-linearly across a task. The cross-layer transfer connecting serving-stack KV retrieval (Quest) to agent long-term memory design offers concrete architectural guidance for building agents whose memory degrades gracefully under budget pressure. The COMPACT-Bench proposal is important for the agent-systems community: no current benchmark measures repeated-compaction error accumulation, so standard single-turn evaluations systematically underestimate the cost agents actually pay. The rate–distortion lens also provides principled criteria (reversibility, query-conditioning) for evaluating agent memory designs such as MemGPT, Mem0, and RAPTOR on a common axis.