Skip to content

TRAP: Targeted Redirecting of Agentic Preferences

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TRAP is a black-box adversarial framework that manipulates VLM-powered agentic systems by injecting semantics into the CLIP embedding space via diffusion-based optimization, causing agents to consistently select an attacker-chosen image without requiring access to model weights or gradients. It combines a Siamese semantic network for identity-preserving optimization with a layout-aware spatial mask decoded through Stable Diffusion, producing visually natural yet semantically steered adversarial images. Across six leading multimodal models, TRAP achieves near-perfect attack success rates, exposing a fundamental cross-modal vulnerability beyond pixel-level robustness.

Problem

Existing adversarial attacks on VLM agents either require white-box access (weights/gradients) or produce visually detectable perturbations, making them impractical against real-world agentic deployments. No prior method addresses stealthy, black-box, semantic-level manipulation of the vision-language selection layer that autonomous agents (e.g., GUI agents, e-commerce recommenders) use to make decisions.

Method

TRAP operates entirely in CLIP's shared image-text embedding space under a strict black-box threat model (query access only). Given a target image and attacker-chosen positive guidance prompt \(p_{pos}\), it optimizes an adversarial image embedding \(e_{adv}\) via a composite loss:

\[L_{total} = \lambda_1 L_{sem} + \lambda_2 L_{dist} + \lambda_3 L_{LPIPS}\]
  • \(L_{sem}\): cosine distance between \(e_{adv}\) and \(e_{pos} = E_T(p_{pos})\), pulling the image embedding toward the target semantic concept.
  • \(L_{dist}\): Siamese network \(S_{dist}\) decomposes embeddings into common and distinctive components; penalizes deviation of \(e_{adv}\)'s distinctive branch from the original, preserving identity while channeling semantic edits through the common branch.
  • \(L_{LPIPS}\): perceptual similarity loss in pixel space, computed after decoding through a layout-aware pipeline: a lightweight MLP generates a soft semantic heatmap \(A\), refined with a DeepLabv3 foreground mask (\(A_{final} = A \odot F_{seg}\)), then the modulated common embedding is decoded via Stable Diffusion v2.1 Img2Img.

Optimization runs for up to 20 outer iterations × 20 inner gradient steps (Adam, lr=0.005). The final \(x_{adv}\) is evaluated over 100 randomized N-way ordering trials to measure selection probability.

Key Contributions

  • TRAP framework: first black-box semantic adversarial attack specifically targeting the vision-language selection layer of agentic AI systems.
  • Siamese embedding decomposition: separates common (prompt-steerable) from distinctive (identity-preserving) features, enabling targeted semantic injection without visual collapse.
  • Layout-aware spatial masking: combines a learned MLP heatmap with DeepLabv3 segmentation to confine edits to semantically relevant foreground regions.
  • Transferability across architectures: adversarial examples crafted via surrogate CLIP transfer to non-contrastive (CogVLM) and fully closed-source (GPT-4o) models.
  • Empirical evidence of semantic vulnerability: demonstrates that existing defenses (Gaussian noise, CIDER, MirrorCheck, adversarial training) do not eliminate the attack.

Results

  • TRAP vs. baselines on COCO 100-sample N-way selection task (Attack Success Rate):
Model SPSA Bandit SA_AET TRAP
LLaVA-1.5-34B 36% 6% 85% 100%
Gemma3-8B 27% 2% 67% 100%
Mistral-3.1-24B 22% 1% 61% 100%
Mistral-3.2-24B 11% 0% 55% 99%
GPT-4o 1% 0% 12% 63%
CogVLM 18% 0% 42% 94%
  • Under defenses (LLaVA-1.5-34B): TRAP + Gaussian Noise → 100%; TRAP + CIDER → 100%; TRAP + MirrorCheck → 100%; on Robust-LLaVA: TRAP + MirrorCheck → 74%.
  • System prompt robustness: mean |∆ASR| ≤ +2% across four rephrased system prompt variants.
  • Temperature robustness: ASR curves nearly identical at T=0.1 and T=0.7 across all models.

Limitations

  • Assumes the agent's selection mechanism relies on contrastive vision-language similarity (cosine scoring in shared embedding space); effectiveness may degrade in architectures that fully abandon contrastive objectives.
  • Success depends on auxiliary component quality (layout mask accuracy, diffusion model fidelity); may degrade on edge-case images or under constrained compute.
  • Computationally expensive: ~520 seconds per iteration vs. 376s (SPSA) and 110s (Bandit) on 4×A100 GPUs; real-time deployment requires distillation or offline amortization.
  • Evaluation is limited to 100 COCO instances; generalization to diverse real-world agentic environments (navigation, booking, coding agents) is not empirically verified.

Relevance to Vision-Language Models

TRAP directly targets the core cross-modal reasoning mechanism of VLMs — the joint CLIP embedding space — showing that semantic alignment, the property that makes VLMs powerful, also makes them uniquely exploitable. For researchers tracking VLMs, this highlights that safety/robustness evaluation must extend beyond pixel-level perturbation budgets to include embedding-space semantic manipulation. The transferability finding (attacks generalize from open CLIP to closed GPT-4o) suggests the vulnerability is structural to the contrastive pre-training paradigm shared across most modern VLMs. This motivates developing embedding-level defenses and semantic robustness criteria as a prerequisite for deploying VLMs in agentic pipelines.