Skip to content

DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection

🕒 Published (v1): 2026-06-25 19:34 UTC · Source: Arxiv · link

Why this paper was selected

Visual memory benchmark for multimodal agents; fills gap between text-memory evals and real embodied perception

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DMV-Bench is the first interactive, multi-session benchmark that isolates long-horizon visual memory in multimodal agents by injecting unique incidental visual cues into product images that are deliberately excluded from all text channels. The authors also propose DualMem, a dual-coding memory architecture that maintains parallel visual (SigLIP-2) and verbal (SBERT) signals, fusing them at retrieval with a tunable weight \(\alpha\). DualMem outperforms all baselines including three recent multimodal memory systems across every tested chain length on both Gemini 2.5 Flash and Qwen2.5-VL-7B.

Problem

Existing agentic memory benchmarks (LoCoMo, LongMemEval, MemoryArena) are text-only or QA-style; even benchmarks with visual observations (VisualWebArena, MemEye) allow agents to solve ostensibly visual tasks via captions or alt-text, and they do not probe whether an unflagged visual detail survives multiple session boundaries. No prior benchmark forces an agent to remember purely visual information encountered incidentally across a long, multi-session, interactive horizon.

Method

Benchmark construction. A synthetic e-commerce storefront serves 1,000 product variants (10 categories × 10 styles × 10 variants). Each product image has a unique incidental cue — a small color–object pair (e.g., "teal sleep mask") rendered directly into the pixel of the product image by Nano-Banana. The L2-leakage contract enforces that no cue identifier appears in any text channel (title, description, alt-text, URL, meta-tags). A task is a chain of \(J\) sessions; a memoryless ReAct agent comparison-shops across categories, encountering cues without instruction to attend to them. Between sessions the in-context window is wiped; only the memory bank persists. A read-only recall agent is then asked to navigate to a cued product ("take me back to the product with the teal sleep mask"), scored by exact URL match.

Recall reach \(r = \text{(recall session)} - \text{(visit session)}\) indexes how many context wipes a cue must survive; sweeping \(r\) yields a retention curve. Evaluation uses a shared-prefix rollout tree to avoid re-running identical early sessions: branching factor \(B\) over depth \(J\) costs \(\frac{B^J - 1}{B - 1}\) runs while generating \(\sim B^{J-1}\) distinct recall paths, giving roughly a \(J\times\) saving.

DualMem. Inspired by dual-coding theory, each observation is encoded into: - Visual signal: \(v_o = \text{SigLIP-2}(\text{image})\) (L2-normalized) - Verbal signal: \(t_o = \text{SBERT}(\text{VLM caption})\) (L2-normalized)

At retrieval, the query is embedded into \(q_v\) and \(q_t\); per-bank-entry scores are computed as inner products \(s_v^{(i)} = \langle q_v, v_i \rangle\) and \(s_t^{(i)} = \langle q_t, t_i \rangle\), then min-max normalized and fused:

\[s^{(i)} = \alpha \hat{s}_v^{(i)} + (1 - \alpha)\hat{s}_t^{(i)}\]

with \(\alpha = 0.75\). The top-scoring entry is injected as both the raw image and its caption.

Key Contributions

  • DMV-Bench: first interactive, multi-session benchmark isolating long-horizon visual agent memory via incidental cue injection and the L2-leakage contract (46,265 recall-probe tasks on Gemini; 18,588 on Qwen).
  • Recall-reach retention diagnostic: probes TSR as a function of session boundaries survived; evaluated efficiently via a shared-prefix rollout tree.
  • DualMem: dual-coding memory architecture with parallel SigLIP-2 visual and SBERT verbal channels fused at retrieval with tunable \(\alpha\).
  • Empirical characterization of asymmetric dual coding: vision carries the incidental cue end-to-end; the verbal channel (\(\alpha \approx 0.25\)) contributes mainly as a query-grounding scaffold rather than a cue carrier.

Results

  • DualMem leads at every \(J \in \{5, 10, 15, 50\}\) on both back-ends (Table 2):
  • Gemini 2.5 Flash: 82.7 / 75.2 / 71.3 / 65.1% vs. next-best M2A at 65.7 / 63.0 / 59.6 / 64.7%
  • Qwen2.5-VL-7B: 81.1 / 77.2 / 75.1 / 68.3% vs. next-best M2A at 70.4 / 64.8 / 62.6 / 58.7%
  • NoMemory and TextOnly achieve 0% everywhere, confirming the L2-leakage contract holds.
  • Caption baseline achieves 58.9–67.3%, demonstrating that when the captioner happens to verbalize the cue (~16.5% of cases), text memory partially works — but degrades sharply with reach.
  • DualMem does not degrade with bank size while baselines do, ruling out a small-bank confound.
  • DualMem is flat across encoding position \(t\) while baselines show position drift.
  • Replacing SigLIP-2 with CLIP costs 11.5 points (76.9 → 65.4).
  • Caption-only injection collapses to 65.1% vs. image+caption at 76.9%, confirming vision is the critical injection channel.

Limitations

  • Benchmark is restricted to a single synthetic domain (furniture e-commerce); real-world visual diversity (cluttered scenes, naturalistic images) is not tested.
  • The incidental cue vocabulary (color–object pairs) is purpose-built and bijective; natural scenes may have ambiguous or non-unique visual cues.
  • Qwen2.5-VL-7B violates the "visit ≥3 categories" instruction in 33.1% of sessions, making its probe counts sparser and its results less reliable than Gemini's.
  • \(\alpha = 0.75\) is tuned empirically on the same benchmark; generalization to other cue types is unverified.
  • The rollout tree evaluation, while efficient, introduces an ordering structure that may not reflect the fully independent multi-session setting.
  • DualMem relies on an external VLM captioner that describes the incidental cue in only 16.5% of cases, suggesting potential for significant improvement via cue-aware captioning.

Relevance to Agentic AI / LLM Agents

DMV-Bench directly addresses a critical gap in agentic memory research: the assumption that text notes are sufficient for cross-session memory fails whenever a user's retrieval cue is a visual detail the agent never flagged as important. This work formalizes the when question — distinguishing tasks that require visual memory from those text suffices for — which is essential for designing memory modules in long-horizon web agents, embodied agents, and personal assistant systems. DualMem's architecture is a drop-in external memory module (not tied to a specific VLM), making the pattern immediately applicable to existing ReAct-style agent frameworks. The asymmetric dual-coding finding (vision carries the cue, text grounds the query) is a concrete empirical principle that should inform how future agentic memory systems allocate encoding budget across modalities.