Skip to content

MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning

🕒 Published (v1): 2025-11-04 18:27 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MemSearcher replaces ReAct's ever-growing history concatenation with a compact, iteratively updated natural-language memory capped at 1,024 tokens, keeping LLM context \(O(1)\) across turns. End-to-end training uses multi-context GRPO, a novel extension of GRPO that propagates trajectory-level advantages to each independent turn. On seven QA benchmarks, MemSearcher outperforms all ReAct-style baselines at matched or smaller model sizes.

Problem

ReAct-based search agents append every thought, action, and retrieval observation to the LLM context, causing context length and FLOPs per turn to grow as \(O(n)\) in the number of turns \(n\). Retrieved passages are noisy and often irrelevant, and cross-turn entity disambiguation degrades. Standard GRPO cannot optimize MemSearcher-style trajectories because each turn has a distinct context \(c_{i} = (q, m_{i-1})\), making vanilla single-context GRPO inapplicable.

Method

At each turn \(i\), the agent receives only the question \(q\) and a compact memory \(m_{i-1}\) (≤1,024 tokens). It produces a reasoning trace and an action (search query or final answer); after the environment returns observation \(o_i\), the LLM acts as a memory manager, overwriting \(m_{i-1}\) with \(m_i\) that retains only question-relevant facts. This keeps context tokens stable regardless of turn count.

Multi-context GRPO extends GRPO to this setting. For a group of \(G\) trajectories, trajectory \(T_i\) contains \(n_i\) turns \(\{T_{i,j}\}\). A single reward \(R_i\) is computed per trajectory and normalized to advantage \(A_i\) within the group (Eq. 4). This advantage is uniformly propagated to every turn: \(A_{i,j} = A_i\). Each turn is then treated as an independent optimization target with its own importance-sampling ratio \(r_{i,j}(\theta) = \pi_\theta(T_{i,j}|c_{i,j}) / \pi_{\theta_\text{old}}(T_{i,j}|c_{i,j})\), and the full objective (Eq. 6) clips and sums over all \((i,j)\) pairs with a KL penalty against a reference policy. A format + F1-score reward function (Eq. 8) supervises the rollout.

Key Contributions

  • MemSearcher framework: LLM doubles as memory manager, replacing history concatenation with a bounded natural-language memory; context complexity drops from \(O(n)\) to \(O(1)\) per turn and total FLOPs from \(O(n^2)\) to \(O(n)\).
  • Multi-context GRPO: first end-to-end RL algorithm for multi-turn agents whose turns have structurally different input contexts; enables annotation-free training.
  • State-of-the-art accuracy on 7 knowledge-acquisition benchmarks across 3B / 7B / 14B model scales, while holding context below 4K tokens throughout interactions.

Results

  • MemSearcher 3B: 43.8 avg EM across 7 datasets — best in the 3B class (next best O2-Searcher 3B: 39.1) and exceeds all 7B baselines (best competing 7B: ReSearch 7B at 43.6).
  • MemSearcher 7B: 48.9 avg EM — beats ReSearch 32B (48.3) at one-quarter the parameters.
  • MemSearcher 14B: 51.7 avg EM — best overall (ReSearch 32B: 48.3; Search-R1 14B-base: 47.8).
  • Context tokens stay below 4K tokens at turn 10; ReSearch 7B exceeds 14K tokens at the same point (Figure 4).
  • RL-trained MemSearcher (43.8) substantially outperforms SFT-distilled equivalent (28.5 avg, Table 4) and the untrained base model (14.4 avg, Table 3).

Limitations

  • Memory is capped at a fixed 1,024-token budget; tasks requiring long-range synthesis across many retrieved passages may force lossy compression.
  • The memory manager and the reasoning agent share the same LLM weights, so degraded memory writing also impairs reasoning — errors are not independently correctable.
  • Experiments use a static 2018 Wikipedia dump with E5 retrieval; generalization to live web search (as used by ZeroSearch / R1-Searcher baselines) is noted as future work.
  • Maximum memory length ablation (Figure 6) is only shown for 7B and 14B; optimal length likely depends on task complexity in ways not fully characterized.
  • Training data is limited to NQ + HotpotQA; out-of-distribution performance may degrade on domains with different passage structures.

Relevance to Harnesses / Meta-Harnesses

MemSearcher is directly relevant as a training harness design problem: multi-context GRPO is a general harness mechanism for end-to-end RL over any multi-turn agent trajectory where turns share a reward but have heterogeneous contexts. The memory-manager pattern — where one model component orchestrates what state persists across steps — is a micro-harness pattern applicable beyond search agents to any tool-using pipeline that must remain context-bounded. For meta-harness builders, the key insight is that trajectory-level reward propagation with per-turn optimization targets decouples the credit assignment problem from the context-isolation problem, enabling harnesses that span arbitrarily many LLM invocations under a single training signal.