Skip to content

Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents

🕒 Published (v1): 2026-07-17 06:05 UTC · Source: Arxiv · link

Why this paper was selected

Black-box memory poisoning attacks on multimodal agents; critical security surface for long-term memory systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LUCID is a black-box adversarial framework that corrupts the persistent multimodal long-term memory of AI agents by applying imperceptible pixel-level perturbations exclusively to user-shared images, with no access to model weights, text channels, or memory indices. It demonstrates two failure modes—memory poisoning and memory injection—achieving 61.6% and 58.4% attack success rates (ASR) respectively across five architecturally diverse memory backends.

Problem

Multimodal AI agents unconditionally trust visual data retrieved from persistent episodic memory: retrieval is governed by embedding similarity, not semantic integrity verification, so a perturbed image that occupies a different region of embedding space is retrieved and acted upon with the same confidence as authentic records. Prior attacks on agent memory require text-channel access, direct write access to the memory store, adversarial captions alongside perturbed images, or trigger prompts—capabilities often unavailable to a realistic attacker. No prior work addressed strictly image-bounded, trigger-free corruption of persistent episodic memory under full black-box conditions across diverse memory architectures.

Method

LUCID operates in three stages under a strict \(\ell_\infty\)-bounded, image-only threat model (\(\varepsilon = 16/255\)):

S1 – Adversarial Target Design. A candidate pool is sampled from ShareGPT4V-100K. Each candidate \(p\) is scored by a composite: $\(S(p; q, a_{gt}) = \alpha\,\Phi_R(p,q) + \beta\,\Phi_{\text{contra}}(p,q) - \gamma\,\Phi_C(p,a_{gt})\)$ with \(\alpha{=}1.0, \beta{=}2.0, \gamma{=}0.5\). \(\Phi_R\) estimates retrieval displacement using CLIP ViT-B/32; \(\Phi_{\text{contra}}\) rewards semantic contradiction of the ground-truth answer (via a local LLaMA-3.1-8B for attribute extraction); \(\Phi_C\) penalizes semantic overlap with the ground truth. The top-scoring candidate becomes the adversarial target \((x^*, c^*)\).

S2 – Payload Construction. Two modes: (i) In-context poisoning replaces the image \(x_j\) of an existing turn whose surrounding text is preserved, creating an internal text–visual inconsistency; (ii) Out-of-context injection plants an adversarial image in a new user turn on a fresh topic with no prior textual grounding, so the memory system stores a fabricated concept with no consistency signal to flag it.

S3 – Adversarial Perturbation. Solves: $\(\max_{\delta:\|\delta\|_\infty \leq \varepsilon} \mathcal{L}(x_s + \delta,\, x^*,\, c^*), \quad \mathcal{L} = \mathcal{L}_{\text{OT}}(x̃, x^*) + \lambda_{\text{txt}}\,\mathcal{L}_{\text{txt}}(x̃, c^*)\)$ \(\mathcal{L}_{\text{OT}}\) is the FOA-Attack objective with global cosine alignment and optimal-transport cluster assignment over \(K{=}10\) feature clusters across a surrogate ensemble {CLIP-ViT-B/16, CLIP-ViT-B/32, LAION-CLIP} with dynamically updated per-model weights. \(\mathcal{L}_{\text{txt}}\) (novel extension) aligns the adversarial image with target caption \(c^*\) in CLIP's joint text-image space, reinforcing that the memory backend's MLLM captions the perturbed image using the target semantics. Optimization runs for \(T{=}1000\) I-FGSM projected sign-gradient steps.

Key Contributions

  • First strictly image-bounded, trigger-free attack on persistent episodic multimodal memory under full black-box conditions (no access to MLLM weights, retrieval encoder, memory indices, or text channel).
  • LUCID framework with a three-stage pipeline: semantically contradictory target selection, dual-mode payload construction (poisoning vs. injection), and FOA-Attack extended with a text-alignment objective.
  • Formalization of two distinct failure modes: in-context poisoning (corrupts a real remembered event) and out-of-context injection (fabricates an event with no detectable text–visual inconsistency).
  • Evaluation across five architecturally diverse backends (MuRAG, NGMemory, AUGUSTUS, UniversalRAG, Mem0Memory) and multiple frontier MLLMs (GPT-4o, GPT-4.1, Claude Haiku 4.5, Gemini Flash 2.5).
  • Demonstration that out-of-context injection is structurally undetectable by consistency-based defenses at storage time.

Results

  • In-context poisoning (Table 2, GPT-4o, avg across 5 backends): 61.6% ASR (Visual Similarity) vs. 67.9% oracle; retrieval rates closely track oracle (e.g., UniversalRAG 89.2% adversarial vs. 92.5% oracle; MuRAG 91.7% vs. 84.5%).
  • Out-of-context injection (Table 3, GPT-4o): 58.4% ASR; AUGUSTUS Cond. ASR 75.0% adversarial vs. 56.3% oracle (adversarial images that are retrieved are more effective at eliciting false responses than the oracle image for this backend).
  • Collateral damage: LLM Judge scores drop 0.12–0.20 on average; sharpest degradation on NGMemory (0.778 → 0.571 Judge) despite lowest adversarial retrieval rate (41.9%), confirming that even partial visual corruption damages vision-critical turns.
  • Attack is architecture-agnostic: succeeds across dense retrieval, graph-structured (NGMemory), context-augmented (AUGUSTUS), mixed-modality (UniversalRAG), and production-scale (Mem0Memory) backends.
  • Adversarial perturbations (\(\varepsilon{=}16/255\)) are imperceptible and transfer from surrogate ensemble to victim encoders/MLLMs with no white-box access.

Limitations

  • \(\varepsilon{=}16/255\) perturbation budget, while standard in adversarial ML, may be detectable by pixel-level forensic tools not evaluated here.
  • Evaluation relies on Mem-Gallery, a benchmark with a mix of visually- and text-grounded questions; text-recoverable answers partially compensate for poisoned images, underestimating true impact in purely visual domains (e.g., medical imaging, navigation).
  • The surrogate ensemble (CLIP-ViT-B/16, ViT-B/32, LAION-CLIP) may not fully cover encoder families used by future production memory systems, potentially limiting transfer.
  • No defenses are proposed or evaluated; the paper establishes the attack surface without providing mitigations.
  • Attack entry points (compromised CDN, malicious SDK, pre-perturbed webpages) assume adversarial control over the image delivery channel, which may not always be feasible.
  • The paper is truncated before full Table 3 results are reported, leaving some injection ASR numbers across all five backends unverified from the provided text.

Relevance to Agentic AI / LLM Agents

This work directly threatens the episodic memory layer that is becoming central to long-horizon, personalized multimodal agents—systems used in health monitoring, personal assistance, and robotics. By showing that the visual modality creates a persistent, architecture-agnostic attack surface that text-based defenses cannot detect, LUCID motivates a fundamental rethink of trust assumptions in multimodal RAG and memory pipelines. The result is particularly acute for agentic systems where memory is shared across sessions: a single poisoned image compounds into corrupted future retrievals, making this a systemic rather than transient vulnerability. This work should drive adoption of semantic integrity verification at memory write time and adversarially robust multimodal encoders as core requirements for production agentic memory systems.