Skip to content

REMem: Reasoning with Episodic Memory in Language Agent

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Episodic memory with spatiotemporal context models human-like recall in LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

REMem is a two-phase episodic memory framework for language agents that represents interaction histories as a hybrid graph of time-aware gists and temporal fact triples, then queries this graph with a ReAct-style agentic retriever. It outperforms Mem0, HippoRAG 2, and Graphiti by 3.4% and 13.4% absolute on episodic recollection and reasoning benchmarks respectively, and is the only method exceeding 90% exact match on the Test of Time benchmark.

Problem

Existing memory systems for language agents—parametric weights, RAG, and graph-based methods like HippoRAG 2 and Mem0—operate under a semantic memory paradigm that discards spatiotemporal context. They lack explicit event modeling with situational dimensions (time, location, participant, emotion) and rely on one-shot similarity-based retrieval that cannot support multi-step logical operations like temporal range filtering, ordinal constraints, or event counting. No prior system provides a unified representation combining episodic recollection (binding situational elements to events) and episodic reasoning (multi-step inference across linked events).

Method

REMem proceeds in two phases:

Indexing: Each utterance or session is processed to extract (1) gists—concise natural-language event summaries with parsed, absolute timestamps capturing participants, actions, objects, locations, and quantities; and (2) facts—\((subject, predicate, object)\) triples with Wikidata-style temporal qualifiers (\(point\_in\_time\), \(start\_time\), \(end\_time\)). These populate a hybrid memory graph: gist nodes connect to their constituent phrase nodes, phrase nodes are directly linked by predicate edges, and synonymy edges are added between gist nodes whose embedding similarity exceeds a threshold of 0.8 (following HippoRAG 2).

Agentic Inference: A ReAct-style agent iterates over the graph using five curated tools: - semantic_retrieve(query, start_time, end_time, ...) — embedding-based (NV-Embed-v2) seed retrieval with optional temporal bounds - lexical_retrieve(...) — BM25-based retrieval with the same temporal filters - find_gist_contexts(gist_id, ...) — expands episode-level narrative context from a seed gist - find_entity_contexts(subject, predicate, object, ..., aggregation) — structured entity lookup supporting filtering, ordering, offset, and aggregation over the fact subgraph - output_answer(answer) — flow-control termination

The agent decomposes complex queries, issues retrieval calls, performs graph exploration with temporal constraints, and terminates when sufficient evidence is gathered. REMem-I uses up to 5 iterative steps; REMem-S performs a single embedding retrieval step.

Key Contributions

  • Formal definition of two progressive episodic memory challenges: episodic recollection (situational binding) and episodic reasoning (multi-step temporal inference).
  • Hybrid memory graph combining gist nodes (context-level, situational) and fact phrase nodes (concept-level, temporally scoped triples), maintaining contradictory entries for historical fidelity.
  • Curated tool suite for agentic inference over the graph, enabling temporal range filtering, graph neighborhood expansion, ordinal constraints, and aggregation beyond flat vector retrieval.
  • Comprehensive 4-benchmark evaluation (LoCoMo, REALTALK, Complex-TR, Test of Time) including refusal behavior on unanswerable questions.

Results

  • Episodic Recollection (LoCoMo LLM-J): REMem-S 77.5% vs. HippoRAG 2 74.0% (best prior), NV-Embed-v2 73.0%; approaches Oracle Message (81.0%).
  • Episodic Recollection (REALTALK LLM-J): REMem-S 65.3%, REMem-I 63.7% vs. HippoRAG 2 55.8%; overall +3.4% absolute improvement.
  • Episodic Reasoning (Complex-TR LLM-J): REMem-I+TISER 92.0% vs. NV-Embed-v2+TISER 88.3%; REMem-I alone 89.6%.
  • Episodic Reasoning (Test of Time EM): REMem-I 93.1% — the only method exceeding 90%; HippoRAG 2 66.9%, Full-Context 79.7%; +13.4% absolute over best prior memory system.
  • Refusal (LoCoMo): REMem F1 64.0% (precision 73.3%, recall 56.8%) vs. Graphiti F1 53.1% and Mem0 F1 13.5%.
  • Token efficiency: REMem-I uses ~9k input tokens per LoCoMo query vs. ~26k for Full-Context.
  • Ablation: Removing gists drops LoCoMo LLM-J from 76.2 to 48.9; removing facts drops Complex-TR LLM-J from 89.6 to 87.2; removing synonymy edges reduces F1/BLEU-1 without large LLM-J impact.

Limitations

  • Relies on GPT-4.1-mini for extraction and QA; indexing quality is LLM-dependent, and cost scales with corpus size.
  • Evaluation is confined to four English-language benchmarks; generalization to other languages or modalities is untested.
  • The agentic inference loop introduces variable latency; the paper does not report end-to-end wall-clock performance.
  • Refusal recall (56.8%) remains below precision (73.3%), meaning the system still generates spurious answers for a substantial fraction of unanswerable queries.
  • The similarity threshold for synonymy edges (0.8) and the maximum inference step counts (3–5) are validated on small held-out sets and may not transfer across domains.

Relevance to Agentic AI / LLM Agents

REMem directly addresses a critical bottleneck in long-horizon agentic systems: the inability of current agents to re-experience and reason over their own interaction histories in a spatiotemporally grounded way. The agentic inference design—a ReAct loop with domain-specific graph tools rather than monolithic RAG retrieval—exemplifies the broader trend of giving agents structured, composable action spaces over external knowledge stores. The hybrid gist+fact graph is a concrete architecture for persistent agent memory that extends beyond session context windows, directly relevant to multi-session personal assistants, task agents, and systems requiring continual adaptation. The demonstrated superiority over Mem0 and Graphiti (two widely deployed memory backends) and the robust refusal behavior make REMem an immediately applicable component for production agentic pipelines.