Skip to content

Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection

🕒 Published (v1): 2026-07-11 14:13 UTC · Source: Arxiv · link

Why this paper was selected

Reversible adversarial examples against VLMs for privacy; novel threat model with practical reversibility

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CloakDiff is the first framework for reversible, imperceptible adversarial examples targeting Vision-Language Models (VLMs) for privacy protection. It combines diffusion-based adversarial guided editing (manipulating both pixel-space CLIP embeddings and latent cross-attention maps) with an Invertible Neural Network (INN) for lossless image recovery. The result protects user images from text-based VLM privacy inference while preserving perceptual quality and enabling exact reconstruction in trusted environments.

Problem

Existing adversarial attacks on VLMs (e.g., Anyattack, AttackVLM) inject high-frequency noise that severely degrades visual quality, making them impractical for real-world privacy protection. Existing Reversible Adversarial Example (RAE) methods are designed for purely visual tasks and cannot be applied to multi-modal settings. No prior method simultaneously achieves (1) cross-model/cross-prompt transferability against VLMs, (2) imperceptible visual quality, and (3) lossless reversibility.

Method

CloakDiff operates in two sequential phases:

Phase 1 — Adversarial Guided Editing. Starting from the original image \(x_\text{orig}\), a Stable Diffusion model is used to produce an adversarial image \(x_\text{adv}\) via guided sampling. Three losses are injected into the score function at each denoising step:

  • Pixel-space guidance \(\mathcal{L}_\text{pixel}\): minimizes cosine similarity between the CLIP embedding of \(x_\text{adv}\) and a target image \(x_\text{tgt}\) (a synthetically generated "shift anchor"), pulling the embedding away from the original semantics.
  • Latent-space guidance \(\mathcal{L}_\text{latent}\): enforces deviation of cross-attention maps \(A^{l,h}_\text{cross}\) between \(x_\text{adv}\) and \(x_\text{orig}\) across all layers and heads, corrupting image–text alignment.
  • Self-attention retention \(\mathcal{L}_\text{retain}\): constrains self-attention maps to remain close to those of \(x_\text{orig}\), preserving global structure and visual fidelity.

The combined adversarial score is: $\(s_\theta^\text{adv}(\tilde{x}_t, t) = s_\theta(\tilde{x}_t, t) - \alpha_2 \sum_k \lambda_k \nabla_{\tilde{x}_t} \mathcal{L}_k(\tilde{x}_t)\)$

Null-Text Optimization (NTO) refines the unconditional text embedding to improve reconstruction fidelity. A custom EDM-Heuristic Sampling (EHS) schedule replaces the default diffusion schedule with a nonlinear, density-controlled time discretization: $\(\bar{\alpha}_t = \frac{1}{\left(1 + \left(u_\text{max} + \frac{t-1}{N}(u_\text{min} - u_\text{max})\right)^{1/\rho}\right)^{2\rho}}\)$ where \(\rho\) controls convexity, concentrating steps at low-noise regions to improve adversarial injection accuracy.

Phase 2 — INN-Based Reversible Steganography. Since \(x_\text{adv}\) is generated rather than perturbed, it has no explicit noise encoding suitable for classical RAE. Instead, \(x_\text{orig}\) is embedded directly into \(x_\text{adv}\) using an Invertible Neural Network \(F_\text{INN}\): $\(x_\text{pro} = x_\text{rae} = F_\text{INN}(x_\text{adv}, x_\text{orig})\)$ Recovery in a trusted environment requires only \(x_\text{rae}\): $\(x_\text{orig}^\text{rev} = F_\text{INN}^{-1}(x_\text{rae})\)$

Key Contributions

  • First framework combining adversarial VLM protection with reversibility (RAE) in a multi-modal setting.
  • Dual-domain adversarial guidance: pixel-space CLIP embedding shift plus latent-space cross-attention manipulation, jointly improving black-box transferability across models and prompts.
  • EDM-Heuristic Sampling (EHS): a theoretically grounded, nonlinear diffusion schedule for adversarial guided sampling, with a monotonicity theorem proving the effect of \(\rho\) on noise level.
  • INN-based steganographic reversibility adapted to generative (non-perturbative) adversarial images, enabling lossless recovery without auxiliary information.

Results

Evaluated on 9 VLMs (BLIP, BLIP-2, InstructBLIP, Flamingo, UniDiffuser, LLaVA, MiniGPT-4, Qwen2.5-VL-7B, InternVL-3-5-8B) across ImageNet (NIPS 2017 adversarial subset) and MS-COCO, against baselines DP-RAE, Anyattack, AttackVLM, AdvDiffVLM:

  • Protection performance (lower caption similarity = stronger protection; all metrics normalized 0–100):
  • On BLIP: CloakDiff-RAE achieves BLEU-1=64.25 vs. DP-RAE=81.12, Anyattack=70.36, AdvDiffVLM=69.72.
  • On UniDiffuser: BLEU-1=37.10 vs. DP-RAE=70.56, Anyattack=52.24, AdvDiffVLM=55.33.
  • On LLaVA: BLEU-1=51.03 vs. DP-RAE=75.18, Anyattack=60.46, AdvDiffVLM=69.32.
  • Visual quality (Table 2): CloakDiff-RAE achieves LPIPS=10.85 and FID=73.09, compared to DP-RAE (LPIPS=14.37, FID=93.27), Anyattack (LPIPS=30.18, FID=159.03), and AdvDiffVLM (LPIPS=20.90, FID=133.38).
  • Recovery quality: SSIM=0.99, PSNR=44.37, RMSE=0.0004 — best or tied-best among all reversible methods.

Limitations

  • Relies on a surrogate diffusion model (Stable Diffusion) and CLIP encoder; transferability to VLMs trained with fundamentally different visual encoders is not fully characterized.
  • Generation pipeline (50 optimization iterations × 20 sampling steps) is computationally expensive compared to feed-forward perturbation methods like Anyattack.
  • The target image \(x_\text{tgt}\) must be specified as a "shift anchor"; the sensitivity of protection quality to target image selection is noted but not systematically ablated in the provided text.
  • Evaluation uses caption-similarity metrics as a proxy for privacy leakage; direct measurement of attribute inference accuracy (e.g., gender/age/location) is not reported in the provided text.
  • INN steganography adds a secondary concealment step that may accumulate slight quality degradation over multiple encode–decode cycles.

Relevance to Vision-Language Models

CloakDiff directly targets the multi-modal alignment mechanisms (cross-attention, CLIP embedding spaces) that underpin modern VLMs, exposing how these same alignment structures can be exploited adversarially to suppress private attribute inference. It advances understanding of VLM robustness by demonstrating that cross-attention map manipulation transfers effectively in black-box settings across architecturally diverse models (query-transformer, cross-attention fusion, LLM-integrated). For researchers tracking VLMs, this work establishes a new attack surface — text-based privacy inference via VLM querying — and provides a concrete defense benchmark against nine representative architectures. The approach also highlights the tension between VLM visual understanding capabilities and user privacy, a concern increasingly relevant as VLMs are deployed in consumer-facing applications.