Skip to content

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

🕒 Published (v1): 2026-05-28 04:27 UTC · Source: Arxiv · link

Why this paper was selected

WorldMemArena: memory must track evolving world state and revise stale entries

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WorldMemArena is a multimodal multi-session benchmark of 461 tasks that evaluates agent memory not as static recall but as a four-stage lifecycle (write → maintain → retrieve → use) embedded in an Action–World Interaction Loop. It is the first benchmark to provide stage-level diagnostic annotations and to run a head-to-head comparison of long-context, RAG, external memory, and harness-based memory agents. Key finding: higher memory quality does not guarantee better downstream QA; harness-based agents are more flexible but remain costly and less reliable.

Problem

Existing memory benchmarks (a) test recall over static dialogue rather than dynamic agent execution, (b) collapse evaluation to final-answer accuracy with no lifecycle breakdown, and (c) convert visual observations to captions, making multimodal evidence underexamined. Crucially, no prior work provides a principled comparison between hand-designed memory pipelines and harnesses that self-author memory, so the relative value of harness-based memory is unknown.

Method

Memory is formalized as an Action–World Interaction Loop: at step \(t\), the agent observes \(o_t = \Omega(z_t)\), acts according to \(a_t = \pi(o_t, m_t)\), and the environment transitions as \((z_{t+1}, f_t) = \mathcal{E}(z_t, a_t)\). A trajectory \(\tau\) is segmented into sessions \(\tau^{(1)} \circ \cdots \circ \tau^{(S)}\); within each session the agent sees only local context while world state persists across sessions.

Four lifecycle stages are made independently evaluable: 1. Write — selective retention: \(\Delta_s = \text{Write}(m_{s-1}, \tau^{(s)})\) 2. Maintain — revision and consolidation: \(m_s = \text{Maintain}(m_{s-1}, \Delta_s)\) 3. Retrieve — decision-relevant evidence: \(R_{s,q} = \text{Retrieve}(m_s, q)\) 4. Use — faithful answer generation: \(\hat{y}_{s,q} = \text{Answer}(q, R_{s,q})\)

The benchmark covers two regimes: Lifelong Evolution (evolving personal/project states across sessions, 6 domains) and Agentic Execution (real agent trajectories from GUI and embodied environments). Sessions are annotated with gold memory points, state-update markers, distractors, and per-question evidence chains. Three system classes are evaluated: long-context agents (GPT-5.4-mini, Qwen3.5, Gemini 3, DeepSeek V4, Claude Haiku 4.5), manually designed systems (RAG via UniversalRAG, external memory via MemGPT/A-Mem/MIRIX/etc.), and harness-based agents (OpenClaw + Codex paired with GPT-5.4 or DeepSeek-V4).

Key Contributions

  • Formalization of multimodal agent memory as an Action–World Interaction Loop with a four-stage lifecycle that is architecture-agnostic and independently evaluable.
  • WorldMemArena: 461 multi-session tasks, 24,258 QA pairs, 15,595 images, average 18.4 sessions and ~9.1K tokens per sample, spanning dialogue and agentic interaction.
  • Stage-level annotations (gold memory points, state updates, distractors, evidence chains) enabling write/maintain/retrieve/use diagnosis rather than aggregate accuracy.
  • First unified evaluation of long-context, RAG, external memory, and harness-based memory paradigms on a shared benchmark.

Results

  • Best QA-C: MemGPT (57.81%), followed by A-Mem (54.63%); RAG Qwen3-VL-Embedding achieves 51.86%.
  • Best Retrieval Coverage (RC): MemGPT (84.99%), RAG Qwen3-VL-Embedding (73.44%), A-Mem (74.19%).
  • Memory quality ≠ QA quality: Qwen3-VL-Embedding and M2A achieve high memory recall (86.22%, 86.83%) and correctness (98.15%, 97.47%) yet QA-C is only 51.86% and 50.14%, respectively — lower than MemGPT which has lower recall.
  • Multimodal systems underperform text-based ones: ViLoMem and MIRIX have multimodal inputs but post QA-C of 49.77% and 44.46%, while text-only MemGPT reaches 57.81%.
  • Harness-based agents (OpenClaw, Codex) are more flexible and better at adapting to diverse tasks but are harder to decompose (lifecycle stages not individually measurable) and remain costly and less reliable than structured manual systems.
  • Systems degrade on Agentic Execution relative to Lifelong Evolution; performance also varies substantially across domains.

Limitations

  • Harness-based agents can only be evaluated end-to-end because their internal memory state is not exposed, making stage-level diagnosis impossible for the most agent-native paradigm.
  • Lifelong Evolution tasks are partially synthetic (world states generated from templates), which may not capture all real-world complexity.
  • Multimodal evaluation relies on LLM-as-a-Judge with GPT-5.4-mini as evaluator, introducing potential bias.
  • At ~9.1K tokens/sample with 18.4 sessions, costs for harness-based agents are not reported quantitatively.
  • The paper is limited to memory within a single agent; multi-agent shared memory is not addressed.

Relevance to Harnesses / Meta-Harnesses

WorldMemArena is among the first benchmarks to treat harness-based memory (agents like OpenClaw and Codex that self-author and reorganize memory during execution) as a first-class experimental condition, directly operationalizing the question of whether emergent self-management beats hand-engineered pipelines. The central finding — harness memory is more flexible but costlier and less reliable — gives practitioners concrete, empirically-grounded guidance on when to invest in harness-authored memory versus structured RAG or external memory modules. The four-stage lifecycle framework (write/maintain/retrieve/use) is itself a meta-level decomposition that can guide harness designers in instrumenting and auditing each stage of their own pipelines. The inability to decompose harness internals into the four stages also flags a tooling gap: meta-harnesses need introspection APIs that expose intermediate memory states for this kind of evaluation.