Skip to content

AdvEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AdvEDM is a gray-box adversarial attack framework targeting VLM-based embodied decision-making (EDM) systems that selectively corrupts only the perception of a few key objects while preserving the semantics of remaining image regions. This fine-grained control keeps the VLM's chain-of-thought reasoning internally consistent, producing valid-but-incorrect decisions (e.g., "accelerate" when a bus is ahead) rather than the error outputs generated by prior attacks. Two variants—AdvEDM-R (semantic removal) and AdvEDM-A (semantic addition)—achieve 70–84% ASR on autonomous driving benchmarks, dramatically outperforming baselines.

Problem

Existing gray-box adversarial attacks on VLMs (e.g., AttackVLM, CLIP-based PGD) disrupt the global image embedding, which conflicts with the task context encoded in system prompts. In CoT-driven EDM pipelines this misalignment causes the VLM to output invalid responses (error messages, refusals) rather than actionable but wrong decisions, making such attacks inconsequential in the physical world. White-box attacks require full LLM access, which is impractical for proprietary task-specific EDM models.

Method

AdvEDM operates in the gray-box setting, using only the vision-text encoder (e.g., CLIP) as a surrogate.

Region selection: Cosine similarity between ViT patch token embeddings and the text embedding of a target object identifies which patches carry that object's semantics. AdvEDM-R masks patches with high similarity (rich in target semantics); AdvEDM-A selects contiguous background patches with low similarity to foreground objects.

AdvEDM-R (Semantic Removal): Optimizes adversarial perturbation with three losses: (1) push the [CLS] embedding away from the target object's text embedding (global); (2) align masked patches toward a zero-pixel reference image (local erasure); (3) attention-[patch] fixation—constrain the product of attention weights and patch embeddings for non-target patches to match the clean image, preserving remaining semantics.

AdvEDM-A (Semantic Addition): Uses a text-to-image generated reference image R of the target object. Reallocates ViT attention weights toward selected injection patches (scaled from R's attention weights). Aligns global [CLS] to a weighted fusion of clean and reference [CLS] embeddings; aligns local key features at injection sites to those of R. Fixation loss preserves non-injection-region semantics using the reallocated attention map.

Both variants share the same weighted three-term loss under an ℓ₂ perturbation budget ε = 8/255.

Key Contributions

  • Fine-grained adversarial framework for VLM-based EDM systems that outputs valid yet incorrect decisions, enabling real-world physical harm.
  • Cross-modality patch selection strategy using cosine similarity between ViT patch tokens and CLIP text embeddings to localize target-object regions.
  • Attention-[patch] fixation regularizer that preserves semantics of non-target regions during adversarial optimization.
  • Two attack variants: AdvEDM-R (semantic removal) and AdvEDM-A (semantic addition), demonstrated on autonomous driving and robotic manipulation.

Results

General image description (MS-COCO, 5 VLMs): - AdvEDM-R: 76.8% ASR, 66.7% SPR, SS 0.683 - AdvEDM-A: 70.2% ASR, 71.6% SPR, SS 0.726 - Baselines (MF-ii best): 85.2% ASR but only 11.3% SPR — high ASR but catastrophic semantic disruption

Autonomous driving (Dolphins Benchmark): - AdvEDM-R: 74.5% avg. ASR vs. baseline max of 17.3% (PGD) - AdvEDM-A: 71.5% avg. ASR

Autonomous driving (DriveLM-nuScenes): - AdvEDM-R: 80.8% avg. ASR; AdvEDM-A: 76.3% avg. ASR vs. baseline max ~20%

Robotic manipulation (100 real images, 4 VLMs): - AdvEDM-R: 68.8% ASR; AdvEDM-A: 64.5% ASR vs. baseline max of 12.5%

Limitations

  • Requires access to the vision-text encoder (gray-box assumption); transferability to fully black-box settings is noted in appendix but not the primary focus.
  • AdvEDM-A requires a text-to-image generated reference image of the target object, introducing a dependency on external generative models.
  • SPR metric cannot be computed for EDM tasks (VLMs do not enumerate all objects in output), limiting evaluation granularity there.
  • Experiments on physical robotic manipulation use only 100 images from a single setup (UR arm desktop scenario), limiting generalization claims.
  • The threshold Îľ for patch masking and other hyperparameters (w1–w3, β, m) are manually tuned per setting.

Relevance to Vision-Language Models

This paper directly probes a critical failure mode of VLMs deployed as reasoning agents: that coarse adversarial perturbations fail precisely because VLMs are good at contextual consistency checking. The insight that fine-grained, semantically coherent perturbations are necessary to fool CoT-based VLM pipelines is important for understanding robustness boundaries of models like LLaVA, MiniGPT-4, and Dolphins. For the VLM-tracking researcher, this work highlights that CLIP-encoder-level vulnerabilities propagate through downstream task-specific fine-tuning, and that the same cross-modal alignment properties that make VLMs powerful (patch-text similarity) also create exploitable attack surfaces. It motivates defenses at the encoder level (e.g., patch-level certified robustness) rather than only at the LLM decoding stage.