Skip to content

Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents

🕒 Published (v1): 2026-07-06 23:16 UTC · Source: Arxiv · link

Why this paper was selected

Novel architecture: memory inside the reasoning loop, read/written every step — changes how to design agent working memory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current LLM agent designs treat external memory as a once-per-turn resource due to the latency cost of networked vector stores (50–200 ms), but an in-process store answering in ~100 µs collapses that cost by three orders of magnitude, making per-step retrieval feasible. The paper demonstrates causally—holding a fixed per-turn memory budget and varying only store latency—that redundant actions rise monotonically from 0.0/12 at in-process speed to 7.2/12 at 110 ms (exact permutation \(p=0.0079\)). End-to-end, recall improves from 0/5 on all baseline runs to 3.6–4.8/5 with in-loop memory across a GPT-5 model ladder under bounded context windows.

Problem

Networked vector stores impose a per-step tax of \(S \cdot \text{RTT}\) on agent reasoning loops, forcing "memory-first" designs that ration retrieval to once per turn (e.g., Mem0) or require serving-layer scheduling to hide the cost (e.g., SearchAgent-X). This prevents external memory from participating continuously in agent reasoning. Context windows do not substitute: they impose a recurring compute cost ("rent") on every retained token, suffer the lost-in-the-middle findability failure, and fail abruptly once the fact span exceeds window size. The field lacks a substrate-level answer to whether per-step retrieval is architecturally viable.

Method

The core claim is formalized via a cost model and a feasibility frontier. End-to-end latency is:

\[\text{E2E} \approx \sum_{\text{steps}} \bigl(t_{\text{reason}} + f \cdot (t_{\text{embed}} + t_{\text{store}})\bigr)\]

where \(f\) is per-step retrieval frequency. The affordable per-step frequency under a budget share \(\beta\) is:

\[f_{\max} = \frac{\beta}{1-\beta} \cdot \frac{r}{c}\]

with \(r\) = per-step reasoning time, \(c = t_{\text{embed}} + t_{\text{store}}\). At \(\beta=0.1\), \(r=1\,\text{s}\): an in-process store with local embedder (\(c \approx 116\,\mu\text{s}\)) affords \(f_{\max} \approx 953\); a network embedder drops this to 0.55; a cloud store with network embedder gives 0.36—all below the \(f=1\) viability cliff.

Primitive: WorkingMemory wraps an in-memory vector store exposing add, recall(query, k), match(text, threshold), and close. All store ops are instrumented separately from embedding latency.

Task 1 (trip planning): Six-turn conversation in which five durable constraints are stated early, then recalled. A windowed session (\(w=4\) messages) forces constraints out of context. Four conditions: Baseline (no tools, no awareness), Aware (told about the short window), Notes (instructed to restate all constraints at every reply), Memory (real tool-calling via OpenAI Agents SDK backed by WorkingMemory). Models: gpt-5-nano, gpt-5-mini, gpt-5, gpt-5.5; five repeats per condition.

Task 2 (causal loop guard): An agent triages 24 security alerts across 4 turns (6/turn), half being semantic duplicates; before each INVESTIGATE the store is queried under a fixed per-turn budget \(B=100\,\text{ms}\). Store latency is varied by injecting delays (\(+\)1, 5, 15, 30, 50, 110 ms) while keeping store answers identical and correct in all arms. A scripted deterministic version (no LLM) and a real-LLM version (gpt-5-nano, gpt-5-mini) are both run. A third arm implements per-turn RAG as the industry baseline. Match threshold for the guard is calibrated offline (F1 = 1.0 at threshold 0.96 on 20 held-out seeds, using local semantic embedder potion-retrieval-32M at ~32 µs). Primary pre-designated contrast: arm (a) in-process vs. 110 ms rung, exact two-sided permutation test over all \(\binom{10}{5}=252\) partitions.

The extended-mind thesis (Clark & Chalmers 1998) is operationalized as an engineering criterion: the parity principle's criteria of "constant availability" and "direct accessibility without difficulty" are read as a latency budget; endorsement is a wiring property of the loop, not the store.

Key Contributions

  • Establishes retrieval frequency (per-turn vs. per-step) as an agent-level design axis, not just a serving-layer knob, and shows store latency is what gates it.
  • Operationalizes the Clark–Chalmers parity principle as a latency budget: an in-process store (~100 µs) clears the threshold that a networked store (~100 ms) does not, making it eligible as constitutive working memory when the loop is wired to consult it.
  • Provides the first causal evidence that store latency determines task outcome independent of answer quality: under a fixed per-turn budget, the scripted loop guard flips from 0 to 10/10 redundant actions, and the LLM guard traces a monotone dose-response from 0.0 to 7.2/12 redundant actions (\(p=0.0079\)).
  • Demonstrates the regime end-to-end: recall 0/5 on all 40 baseline runs → 3.6–4.8/5 with in-loop memory; 244/244 writes retained; all misses attributed to the agent's single bounded read policy, not the store.
  • Relocates the dominant residual bottleneck from the store to network embedding (~200–400 ms); a local embedder (potion-retrieval-32M) returns the complete operation to ~40 µs.

Results

  • Causal loop guard: 0.0/12 redundant actions at in-process speed → 1.4–1.6 at +15 ms → 7.2/12 at +110 ms (both gpt-5-nano and gpt-5-mini; five seeded workloads per rung; exact permutation \(p=0.0079\); zero guard errors across all runs). Scripted deterministic version: 0/10 at in-process → 10/10 at 110 ms (B=100 ms budget).
  • Budget sweep: Even at B=500 ms, the 110 ms rung leaks 1.6/12 redundant actions; no rung is lossless except in-process.
  • Trip task recall: 0/5 on all 40 baseline and window-aware runs; 3.6–4.8/5 with in-loop memory tools across four GPT-5 models; Notes/restate baseline reaches 5/5 on this five-fact task but costs per-turn token rent growing with working set size.
  • Store latency: Live ops at p50 80–165 µs; with local embedder, complete op at ~40 µs.
  • Feasibility: \(f_{\max} \approx 953\) retrievals/step (in-process + local embedder) vs. 0.55 (in-process + network embedder) vs. 0.36 (cloud store + network embedder) at \(\beta=0.1\), \(r=1\,\text{s}\).
  • Memory reliability: 244/244 writes retained across all runs; every observed recall miss traces to the agent's fixed \(k=8\) single bounded read, not to store loss.

Limitations

  • Task scope is narrow: a single five-fact trip-planning scenario; generalization to larger working sets or more complex tasks is asserted but not measured.
  • Read policy is not optimized: every miss is attributed to a single-read-per-run policy (\(k=8\)), but no adaptive read policy is tested; "memory in the loop" is only partially realized on the read side in the trip task.
  • The Notes/restate baseline outperforms memory tools on the five-fact task, undermining the practical claim at small scale; the paper argues this reverses at larger working sets but does not measure it.
  • Models are restricted to OpenAI GPT-5 variants; no open-weight model evaluation.
  • Ephemeral store only—no cross-session persistence or memory management (retention policy, forgetting) is addressed; authors explicitly flag this as future work.
  • The causal result is conditioned on \(B=100\,\text{ms}\); the budget sweep is included but each choice is contestable.
  • Embedding API as a confound: the main trip-task experiments use network embedding (~200–400 ms), which dwarfs store latency and may mask other effects; the local-embedder result is demonstrated separately rather than as the primary condition.

Relevance to Agentic AI / LLM Agents

This paper directly reframes a foundational assumption in agent memory design—that retrieval belongs outside the reasoning loop—by grounding the argument in a concrete latency model with causal experimental evidence. For practitioners building long-running or multi-step agents (loop guards, deduplication, per-step grounding), the feasibility frontier provides a principled criterion for choosing memory substrate rather than defaulting to cloud vector DBs. The result that every observed recall failure is a read-policy failure, not a write/storage failure, redirects the open research question from "how do we store facts reliably" to "how should agents decide what and when to retrieve"—a design space (retention policy, read scheduling, adaptive \(k\)) that is largely unaddressed. The causal loop-guard experiment is particularly relevant to multi-step tool-use agents that risk action repetition across turns with bounded context.