Skip to content

Towards Building Model/Prompt-Transferable Attackers against Large Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing adversarial attacks on Large Vision-Language Models (LVLMs) overfit to the specific model and prompt they were crafted for, failing at black-box transfer. This paper proposes an information-theoretic attack that disentangles mutual information (MI) between adversarial/benign image patterns and model outputs, using explicit MI constraints during optimization to produce perturbations that transfer across both unseen LVLM architectures and semantically diverse prompts within a single generation pass.

Problem

Prior LVLM adversarial attacks (e.g., PGD, CroPA, UniAtt) optimize perturbations against a specific model and prompt, so the generated adversarial examples fail when transferred to different architectures or different task-aware prompts. Cross-prompt methods like CroPA require complex multi-prompt joint training and still do not address cross-model transfer; no existing method simultaneously achieves both model- and prompt-transferability efficiently.

Method

The core insight is that the mixed mutual information between an adversarial image \(x_v^{adv}\) and LVLM logits \(L'\) linearly decomposes (Theorem 1 + Assumption 1) into benign MI \(I(x_v; L')\) and adversarial MI \(I(\Delta; L')\): $\(I(x_v^{adv}; L') \approx I(x_v; L') + I(\Delta; L')\)$

Two lightweight MI estimator networks are trained separately using the Donsker-Varadhan (KL-divergence) lower bound via Deep InfoMax: one for adversarial MI (encoder \(C_{\psi_A}\) + discriminator \(T_{\omega_A}\)), one for benign MI (\(C_{\psi_B}\), \(T_{\omega_B}\)). Each estimator is trained with a joint max-min objective: the adversarial MI estimator maximizes \(I(\Delta; L')\) (targeted output) while minimizing \(I(\Delta; L)\) (benign output), and vice versa for the benign estimator.

During adversarial perturbation generation, the optimization jointly: 1. Minimizes cross-entropy loss \(l_{ce}\) toward the target text \(y_{tar}\) 2. Maximizes the MI loss \(l_{mi}\) = adversarial MI − benign MI, forcing the LVLM to attend to the perturbation pattern rather than the benign image content

Total loss: \(J = w_1 \cdot l_{ce} - w_2 \cdot l_{mi}\), with \(w_1 = w_2 = 1\), \(\epsilon = 16/255\) (\(\ell_\infty\)), 1000 epochs.

Key Contributions

  • First unified attack achieving simultaneous model- and prompt-transferability for LVLMs within a single adversarial optimization.
  • Information-theoretic decomposition of mixed MI into disentangled benign and adversarial components, with formal proof that the approximation holds under a bijection assumption.
  • Lightweight DV-based MI estimators (2-layer CNN encoder + attention discriminator) that can be trained on PGD-initialized adversarial examples and reused for perturbation generation.
  • Empirical validation showing higher adversarial MI correlates directly with higher transferability across diverse LVLM architectures and prompts.

Results

  • Cross-model transfer (Table 1): On DALL-E dataset, transferring from LLaVA-1.5 to MiniGPT-4, the method achieves SS=0.693 / EM=63.5% / CC=66.9% vs. UniAtt's SS=0.267 / EM=16.2% / CC=23.8% and CroPA's SS=0.093 / EM=0% / CC=0%.
  • Cross-model transfer (Table 1): Transferring from BLIP-2 to InstructBLIP: SS=0.506 / EM=41.2% vs. UniAtt's SS=0.275 / EM=13.4%.
  • Cross-prompt transfer (Table 2): With 60 prompts on DALL-E/LLaVA-1.5, method achieves SS=0.730 vs. UniAtt=0.714 and CroPA=0.711.
  • Transfer to closed-source models (Table 3): Adversarial examples from MiniGPT-4 transferred to GPT-4o achieve SS=0.608 / EM=44.7% / CC=51.3%; UniAtt achieves SS=0.142 / EM=4.1%.
  • GPU efficiency (Table 4): Attack generation takes 9 min / 18.2 GB vs. UniAtt's 294 min / 57.5 GB, while outperforming it on transfer.
  • Defense robustness: Method maintains higher semantic similarity than baselines under Randomization, JPEG Compression, and Diffusion Restoration defenses.
  • Ablation (Figure 3c): Removing either MI constraint degrades transfer performance; both components are necessary.

Limitations

  • White-box access to at least one source model is required to generate the MI estimators and perturbations; purely query-based black-box attack is not explored.
  • MI estimators are initialized using PGD-generated adversarial examples, creating a bootstrapping dependency on an existing attack.
  • Experiments use open-source models from 2023–2024 (LLaVA-1.5, MiniGPT-4, BLIP-2, InstructBLIP); transferability to more recent, larger-scale VLMs (beyond the limited GPT-4o/Claude-3.5 spot-check) is not systematically evaluated.
  • The bijection assumption (Assumption 1) required for MI decomposition is an approximation whose validity in high-dimensional image spaces is justified empirically and via appendix proofs but not guaranteed in general.
  • The disentanglement theorem assumes negligible interaction MI \(I(x_v; \Delta; L')\), which is asserted without a formal bound.

Relevance to Vision-Language Models

This paper directly exposes a practical security vulnerability in the multimodal integration layer of LVLMs: because visual encoders and language models are jointly conditioned, adversarial perturbations can exploit this coupling to hijack model reasoning in a model-agnostic way. The information-theoretic framing—measuring how much the model's output depends on perturbation vs. benign content—provides a principled diagnostic for LVLM robustness that complements standard attack/defense benchmarks. For researchers tracking VLMs, this work establishes that cross-architecture transferability is achievable with moderate compute, raising the practical risk bar for deployed multimodal systems and motivating adversarially robust LVLM training and pre-processing defenses that go beyond single-model hardening.