MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

🕒 Published (v1): 2026-06-29 05:07 UTC · Source: Arxiv · link

Why this paper was selected

Surprising finding on recoverable visual data after memory deletion; security implications

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MemLeak introduces the Information Provenance Graph (IPG), a taxonomy classifying agent memory representations by deletion affordance, and shows that after fact-level deletion, retained user images enable 12.0% fact recovery in multimodal agents (blind baseline 0.0%), with 47% of image leaks invisible to text-only auditing. Content-aware semantic deletion reduces the image residual to 2.0% but at 21.0% retained-image cost.

Problem

When multimodal AI agents are asked to forget a fact, current memory systems delete the text entry but the fact remains recoverable from retained user images via the VLM's implicit visual inference. No existing taxonomy tracks how a single fact propagates across heterogeneous memory layers (text entries, text embeddings, image files, image embeddings, graph edges, implicit visual features, derived facts, behavioral patterns) or classifies each representation by how deletable it is. Provenance tracking alone cannot address cross-fact visual correlation because implicit visual features are never registered as explicit storage operations.

Method

The paper introduces the Information Provenance Graph (IPG), a DAG \(G_f = (V_f, E_f, \delta)\) where \(V_f\) are representation nodes across all storage layers, \(E_f\) captures information flow, and \(\delta: V_f \to \{\text{addr}, \text{link}, \text{unreach}\}\) assigns deletion affordance. addr nodes are directly deletable by ID; link nodes require provenance tracking; unreach (persistent) nodes are not targeted by storage-level deletion and remain as potential leaks. The forgetting completeness metric \(FC_S(f)\) is the fraction of \(V_f\) nodes successfully deleted.

MemLeak benchmark: 113 synthetic entity profiles (20 facts each across 5 categories), with 38 multimodal profiles augmented by 536 Gemini 3.1 Flash-generated images at three visual grounding levels (explicit, implicit, absent). Five probe types target specific IPG layers; the main result uses Probe 1 (text-retract + image-probe). Leakage is determined by a 3-model LLM judge ensemble (GPT-4.1, Claude Sonnet 4, Gemini 2.5 Flash) with majority vote. Systems evaluated: Long-context (behavioral suppression), Naive (text-delete), Mem0 (semantic search deletion), Letta (simulated agent-decided), Oracle (provenance cascade). A content-aware semantic deletion policy flags and removes retained images correlated with the deleted fact.

Key Contributions

• IPG taxonomy classifying agent memory representations by deletion affordance (addr/link/unreach), enabling analysis of why storage-level deletion fails
• Empirical evidence that fact-level erasure fails through multiple channels: retained correlated text (18.3%), retained images (12.0%), and their combination
• MemLeak benchmark decomposing these leakage channels with controlled synthetic data and validated by dual-annotator human judgment (\(\kappa = 0.88\))
• Demonstration that content-aware semantic deletion reduces the image residual to 2.0%, but cannot eliminate it entirely — the remaining cases involve visual cues too diffuse for current VLM auditors to detect

Results

• Text-only direct probing: all deletion-capable systems achieve <1% leakage; behavioral suppression (no deletion) leaks 17.0%
• Retained-image recovery: 12.0% (36/300, 95% CI: [8.8, 16.2]) — blind baseline 0.0%, negative FPR 0.3%
• Retained-text inference (no images): 18.3% (55/300, 95% CI: [14.4, 23.1])
• Channel overlap: 47% of image-leaked facts (17/36) are not recovered by the text-only baseline — images provide a unique, harder-to-audit channel
• Same-fact recovery (upper bound): 48.7%
• Model generality: GPT-4o yields 9.3% (overlapping CI with Claude Sonnet 4's 12.0%); union rate is 16.0%
• Mem0 end-to-end: 16.3% (combines visual inference with text-retrieval residue)
• Real Unsplash photos: 10.6% (30/282), consistent with synthetic 12.0% — not a generation artifact
• Semantic deletion: reduces residual to 2.0% (95% CI: [0.9, 4.3]) at 21.0% retained-image cost
• Per-category: Location (16.7%) and possession (15.2%) highest; semantic deletion reduces all categories uniformly to 1.5–3.4%

Limitations

• The 12.0% residual is an empirical lower bound under the "delete text + tagged images" policy, not a claim of architectural impossibility
• Real photos are stock-photo distribution (Unsplash), not in-situ user data — generalization to authentic user photo collections is open
• Content-aware deletion costs 21.0% of retained images, a significant practical tradeoff for deployment
• Remaining 2.0% involves visual cues too diffuse for current VLM auditors to detect — could be leakage or model capability
• Results are policy-dependent; more aggressive policies shrink the residual but may over-delete retained memories
• Synthetic profiles may not capture all real-world cross-fact visual correlation patterns
• Mem0 end-to-end results (16.3%) conflate visual inference with possible text-retrieval residue and cannot be fully disentangled

Relevance to Vision-Language Models

This paper identifies a fundamental tension in VLM-based agents: the same visual understanding capability that makes VLMs powerful (encoding implicit features like geolocation, architectural style, cultural context from images) creates a privacy liability when those features reconstruct deleted user facts from retained images. The finding that 72% of image-leaked facts require visual evidence beyond retained text confirms that images open a distinct, harder-to-audit leakage channel that text-only privacy audits miss. For anyone building or deploying VLM agents, this work provides a concrete benchmark for measuring the gap between record-level deletion and behavioral forgetting in multimodal settings.