Skip to content

Transferable and Stealthy Adversarial Attacks on Large Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; progressive semantic infusion yields transferable stealthy LVLM attacks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Progressive Semantic Infusion (PSI) is a diffusion-based adversarial attack that crafts stealthy, transferable adversarial images against black-box VLMs by jointly optimizing for feature alignment and naturalness (adherence to the natural image distribution). PSI uses DDPM priors to stay on the natural image manifold, progressive localized alignment objectives to avoid surrogate overfitting, and source-aware denoising to preserve visual fidelity. It outperforms prior methods on both attack success rate and stealthiness across open-source, adversarially trained, and commercial VLMs including GPT-5 and Grok-4.

Problem

Transfer-based adversarial attacks on black-box VLMs suffer from two failure modes: (1) fixed alignment objectives in pixel space push adversarial examples off the natural image manifold, causing poor transferability, and (2) unrestricted attacks (e.g., AnyAttack) introduce perceptible or model-detectable artifacts, failing the stealthiness requirement. No prior method jointly and principally optimizes both transferability and stealthiness.

Method

PSI formulates a joint objective \(\mathcal{L}_{\text{joint}} = p_F(f^{\text{tar}} | x^{\text{adv}}) \cdot p_D(x^{\text{adv}})\) combining alignment on a surrogate \(F\) with naturalness under the data distribution \(p_D\).

Three components address optimization challenges:

  1. Diffusion-based optimization framework: PSI inverts the source image \(x\) to an intermediate latent \(x_{t^*}\) via DDPM inversion (\(t^* = 20\%\) of total steps), then injects alignment perturbations at each denoising step \(t = t^*, \ldots, 1\) via \(x_{t-1} = \text{Denoise}_t(x_t) + \text{Perturbation}(t)\), where each denoising step implicitly optimizes naturalness via the score function \(\mu(x_t, t) \approx x_t + \sigma_t^2 \nabla_{x_t} \log p_D(x_t)\).

  2. Progressive alignment objectives: Instead of a single fixed global cosine similarity, PSI uses timestep-varying localized objectives \(\mathcal{L}_{\text{align}}(t) = \text{cosine}(F(a_t), F(r_t))\), where \(a_t \subseteq \mu(x_t, t)\) is the best-matching candidate crop and \(r_t\) is a co-evolving reference region that interpolates from a salient object bounding box \(o_t\) to the full target image as \(t \to 1\): \(r_t = \text{Interpolation}(o_t, x^{\text{tar}}, 1 - t/t^*)\). SAM provides object detection; \(N=4\) candidate crops of scale \(s \in [0.4, 0.9]\) are scored by cosine similarity to select \(a_t\).

  3. Source-aware denoising: Instead of standard DDIM, the noise sequence \(\{\hat{\epsilon}_t\}\) is derived from forward noising of the source image \(\hat{x}_t = \sqrt{\bar{\alpha}_t} x + \sqrt{1 - \bar{\alpha}_t} n_t\), embedding source cues into the stochastic term \(\hat{\epsilon}_t = (\hat{x}_{t-1} - \mu(\hat{x}_t, t)) / \sigma_t\). These cues propagate through denoising, maintaining visual similarity to the source without explicit pixel regularization.

Key Contributions

  • Formulation of a joint objective over alignment and naturalness as principled motivation for diffusion-based attack design.
  • Progressive alignment with co-evolving region selection that functions as an ensemble-like regularizer over spatial and temporal diversity, mitigating surrogate overfitting.
  • Source-aware DDPM inversion that embeds source structure into the noise trajectory for stealthiness without GradCAM masking or explicit pixel-norm constraints.
  • Empirical demonstration of transferable stealthy attacks on GPT-5 (S-ASR 62.8%), Grok-4 (S-ASR 71.5%), and Gemini-2.5 Flash (S-ASR 71.5%), plus adversarially trained FARE4 (S-ASR 63.5%).

Results

  • GPT-5: PSI ASR 78.6% / S-ASR 62.8% vs. FOA (prior SOTA) ASR 75.8% / S-ASR 56.5%.
  • Grok-4: PSI S-ASR 71.5% vs. FOA 63.4%.
  • Gemini-2.5 Flash: PSI S-ASR 71.5% vs. FOA 64.3%.
  • Claude-3.5 Sonnet: PSI S-ASR 75.0% vs. FOA 72.7%; Claude-3.5 was the most robust commercial model.
  • FARE4 (adversarially robust): PSI S-ASR 63.5% vs. FOA 51.0%, showing PSI degrades least against adversarial training.
  • Imperceptibility: PSI BRISQUE 22.14, LPIPS 0.192—substantially better than all \(\ell_\infty\)-bounded baselines (FOA: 50.37/0.217) and competitive with AdvDiffVLM (22.59/0.214) while vastly outperforming it on ASR.
  • Defense robustness (GPT-5): Under DiffPure, PSI ASR 34.2% vs. FOA 19.7%; under Gaussian smoothing, PSI S-ASR 56.6% vs. FOA 48.2%.
  • Ablation: Removing progressive alignment causes the largest single-component drop (ASR 22.8%); removing co-evolving selection drops ASR from 78.6% to 71.3%.

Limitations

  • PSI requires a pre-trained diffusion model (Stable Diffusion 2.1) and SAM for object detection, making attack generation significantly more computationally expensive than \(\ell_\infty\)-bounded methods.
  • Requires a single NVIDIA A800 80GB GPU; compute cost per adversarial example is not reported.
  • The stealthiness definition (output-level, LLM-judged) is evaluated using GPT-4o as judge, introducing potential bias and reproducibility concerns tied to API-version drift.
  • Transferability gap remains substantial under DiffPure defense (ASR drops ~44 points), indicating diffusion-purification is an effective countermeasure.
  • Attack is evaluated only in the image captioning task; generalization to other VLM task types (VQA, instruction following) is not demonstrated.
  • \(t^*\) (inversion depth) is fixed at 20%; the trade-off between naturalness and alignment suppression is not fully characterized across model families.

Relevance to Vision-Language Models

PSI directly probes the security of state-of-the-art VLMs—including GPT-5, Grok-4, and Gemini-2.5—revealing that commercial safety measures such as adversarial input detection are circumventable when attacks preserve natural image statistics. The finding that naturalness (staying on the data manifold) is a key driver of cross-model transferability provides a mechanistic explanation for why black-box VLMs sharing training distribution with the surrogate are vulnerable to in-distribution perturbations. The introduced S-ASR metric, which requires both semantic manipulation and model-undetectable outputs, sets a more realistic threat model for evaluating VLM robustness than ASR alone. This work motivates a new defense direction: robustness against unrestricted, distribution-aligned perturbations rather than only \(\ell_\infty\)-bounded noise.