MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems¶

🕒 Published (v1): 2026-05-18 15:43 UTC · Source: Arxiv · link

Why this paper was selected

MINTEval: evaluates memory under multi-target interference in long-horizon agent systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MINTEval is an analytical benchmark for evaluating memory-augmented agents on long-horizon, interference-heavy tasks where information is repeatedly revised and must be recalled or aggregated across time. It contains 15.6k QA pairs over contexts averaging 138.8k tokens (up to 1.8M), spanning four domains and five question types. All seven evaluated systems struggle badly, averaging 27.9% accuracy, exposing fundamental gaps in memory construction and retrieval under temporal conflict.

Problem¶

Existing memory benchmarks evaluate recall over static, largely independent events; they do not model the dense temporal interference that arises when real-world information is repeatedly revised, overwritten, or contradicted. Specific gaps: (1) no existing benchmark combines both interdependent inputs and high-depth interference (≥10 updates), (2) lookback queries requiring recovery of earlier states are absent, (3) multi-target aggregation across evolving updates is underrepresented, and (4) benchmarks are mostly single-domain (dialogue), preventing evaluation of domain generalization.

Method¶

MINTEval constructs long-horizon contexts from four domains, each with distinct update dynamics:

State Tracking (bAbI): symbolic facts updated via sequential overwrites (avg. 42 depth, 0.3k tokens)
Dialogue (HorizonBench): concatenated multi-session conversations tracking implicit user preferences (avg. 142 depth, 274k tokens)
Wiki Revisions: chronologically ordered Wikipedia revision histories with factual edits and contradictions (avg. 99 depth, 195k tokens)
Git Commits: GitHub commit histories with cross-file code evolution (avg. 61 depth, 86k tokens)

Question types form two categories: Single-Target Recall (Simple = retrieve current state; History = retrieve a prior state despite subsequent interference) and Multi-Target Aggregation (Ordering, Counting, Multihop). QA pairs are generated via template-filling from metadata (bAbI, HorizonBench) or by prompting Gemini-3.1-Pro on revision metadata (Wiki, Git), with human validation on 20% of instances (95.6% validity).

Seven systems are evaluated: Full Context, RAG, HippoRAG (graph-structured retrieval), MemAgent (query-specific incremental overwrite on Qwen2.5-14B), AtomMem (CRUD-based RL agent on Qwen3-8B), Mem-α (core/semantic/episodic memory on Qwen3-4B), and SimpleMem (three-stage: semantic compression → online synthesis → intent-aware retrieval). Evaluation uses Exact Match after normalization.

Key Contributions¶

First benchmark combining dense interference (both proactive and retroactive), long-range lookback queries, multi-target aggregation, and multi-domain coverage simultaneously
Four naturalistic domains covering symbolic, conversational, encyclopedic, and code-evolution memory tasks
15.6k QA pairs over contexts averaging 86 updates and 138.8k tokens, scaling to 1.8M tokens
Systematic error decomposition separating memory construction failures (−41.7% accuracy drop) from answering-stage failures (−25.2% drop)
Analysis of agent behavioral biases: insertion-heavy operations (avg. 76.8%) and recency bias

Results¶

Overall average accuracy: 27.9% across all systems; best system (MemAgent) reaches 33.4%
By question type: Simple recall 47.5% > Multi-target aggregation 26.5% > History/lookback 21.0% > Ordering 24.0%
MemAgent achieves 85.7% on bAbI Simple but collapses to 7.5% on HorizonBench Simple, illustrating poor cross-domain generalization
Memory-augmented agents vs. non-memory baselines on bAbI: +9.9% average improvement; on other longer-horizon domains: −3.0% average drop
Error decomposition: memory construction accounts for 41.7% performance loss; answering agent contributes 25.2%
HippoRAG is relatively stable across Simple recall tasks but all RAG methods fail on History and aggregation
SimpleMem with frontier models (Gemini-3.1-Flash-Lite + Gemini-Embedding-001) still fails to saturate the benchmark (specific overall number truncated in provided text)
Performance degrades monotonically as the number of intervening updates between a fact and its query increases

Limitations¶

Evaluation metric is Exact Match only; partial credit for aggregation tasks (e.g., partial orderings) is not captured
HorizonBench uses a candidate-answer multiple-choice fallback, creating a methodological inconsistency with other domains
bAbI contexts are very short (0.3k tokens, 42 depth), making it easier and less representative of the interference challenge
Question generation for Wiki/Git relies on a frontier LLM (Gemini-3.1-Pro); despite human validation, automated biases may persist
Systems are evaluated at fixed chunk granularities (15 facts / 1 session / 1 revision per chunk), which may not reflect optimal operating points
The benchmark covers English and code only; multilingual or polyglot interference settings are unexplored

Relevance to Harnesses / Meta-Harnesses¶

MINTEval is directly relevant as a stress-test harness for the memory subsystem of long-horizon agent pipelines: it formalizes the conditions under which memory-augmented orchestration fails (recency bias, insertion-only updates, retrieval under conflict). For meta-harness designers who chain agents over evolving state — e.g., pipelines that accumulate context across tool calls, multi-session workflows, or RAG-augmented loops — the benchmark's error decomposition (construction vs. retrieval vs. reasoning) provides an actionable diagnostic framework. The finding that memory construction failures dominate (−41.7%) over answering failures (−25.2%) implies that harness designers should invest most in the memory-write and update logic, not in the downstream reasoning agent. The multi-domain design also validates whether a harness's memory layer generalizes across task types, a key property for general-purpose orchestration systems.