On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline¶

🕒 Published (v1): 2026-07-16 13:28 UTC · Source: Arxiv · link

Why this paper was selected

Simplicity-first transferable attack on VLPMs; questions complexity of existing pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Existing transferable adversarial attacks on Vision-Language Pre-trained Models (VLPMs) follow a complex three-stage text→image→text pipeline that inadvertently causes surrogate overfitting and wastes computation. This paper proposes SimVLA, which replaces uni-modal word selection with cross-modal identification, filters image-irrelevant words before image perturbation, and eliminates the redundant third stage entirely. The result is a pipeline that is simultaneously simpler, faster, and more transferable than prior state-of-the-art.

Problem¶

Transferable attacks on VLPMs inherit a three-stage optimization pipeline (BERT-Attack → PGD → BERT-Attack) whose mechanisms are under-justified. Three specific flaws accumulate: (1) word importance in Stage 1 is scored using text-only KL divergence, ignoring the paired image; (2) adversarial text fed into Stage 2 contains image-irrelevant words that inject spurious gradients; (3) Stage 3 repeats text optimization against the surrogate, causing over-fitting to the surrogate's loss landscape and reducing black-box transferability despite lowering white-box similarity scores.

Method¶

SimVLA replaces the standard pipeline with three targeted refinements:

Cross-modal word identification (Stage 1, Step 1). Instead of masking each word and measuring KL divergence against the text embedding, SimVLA computes KL divergence against an image embedding center $f_{\text{center}}(x) = \frac{1}{N_I}\sum_{i=1}^{N_I} f_I(x+\delta_i)$, where $\delta_i \sim \mathcal{U}[-\beta\epsilon_x, \beta\epsilon_x]$. The most influential word is: $$w^* = \arg\max_{t' \in S_{\text{mask}}} \text{KL}(f_{\text{center}}(x) \| f_T(t')).$$

Text semantic abstraction (Stage 2, Step 1). Before generating adversarial images, the Top-$K$ words in the adversarial text $t_{\text{adv}}$ that are least relevant to the image (lowest KL divergence when masked) are removed, yielding $t^R_{\text{adv}}$. The image attack loss then targets only the semantically grounded portion: $$L(x) = -\sum_{j=1}^{N_T} \frac{f_I(x) \cdot f_T((t_j)^R_{\text{adv}})}{\|f_I(x)\|_2 \cdot \|f_T((t_j)^R_{\text{adv}})\|_2}.$$

Stage deletion (Stage 3). The third text-attack stage is removed entirely. Empirical evidence (Table 1) shows that retaining Stage 3 lowers white-box similarity but reduces black-box R@1 transferability (e.g., SGA: 32.39% → SGA w/o Stage 3: 35.71% TR R@1), confirming surrogate overfitting. Image optimization uses MIM-PGIA as an auxiliary optimizer to bias perturbations toward flatter loss regions, but this is adopted off-the-shelf and is not a core contribution.

Key Contributions¶

Formal identification of three stage-specific failure modes in the dominant VLPM attack pipeline, each traced to inappropriate cross-modal interaction or redundant computation.
Cross-modal word identification: substituting image embedding center for text embedding in the KL-based word ranking of BERT-Attack.
Text semantic abstraction: masking-based removal of Top-$K$ image-irrelevant words from adversarial text before image attack.
Stage deletion: empirically validated removal of the third pipeline stage to prevent surrogate overfitting.
Comprehensive evaluation on four datasets (Flickr30k, MSCOCO, and others), three downstream tasks, and four VLPMs (ALBEF, TCL, CLIPViT, CLIPCNN).

Results¶

On Flickr30k text-image retrieval, SimVLA outperforms SOTA baseline SA-AET in R@1 transferability by 8.01%–14.71% across surrogate–target model pairs.
SimVLA consumes only ~35.73% of the wall-clock time and ~46.26% of the peak VRAM of the SA-AET baseline.
Selected cross-model TR R@1 results on Flickr30k (surrogate ALBEF → target CLIPViT): SA-AET 37.81%, SimVLA 52.52% (+14.71 pp).
Surrogate ALBEF → target CLIPCNN TR R@1: SA-AET 45.40%, SimVLA 56.69% (+11.29 pp).
Surrogate CLIPViT → target ALBEF TR R@1: SA-AET 13.87%, SimVLA 22.00% (+8.13 pp).
Stage-deletion ablation (Table 1): removing Stage 3 from SGA alone lifts TR R@1 from 32.39% to 35.71% and IR R@1 from 44.07% to 48.32% (Flickr30k, ALBEF→CLIPViT).

Limitations¶

Evaluation is limited to image-text retrieval and related tasks; generalization to other VLPM downstream tasks (e.g., VQA, captioning) is not fully demonstrated in the available excerpt.
The three refinements are validated on a fixed set of four VLPMs; coverage of more recent large-scale VLMs (e.g., LLaVA, InstructBLIP) is not assessed.
The image embedding center neighborhood sampling introduces a hyperparameter $\beta$ whose sensitivity is only partially analyzed.
MIM-PGIA is adopted as a black-box optimizer component, meaning Stage 2 efficiency depends on a third-party method whose own assumptions may not always hold.
White-box attack success rates are lower for SimVLA in some configurations (e.g., ALBEF white-box TR R@1: 94.79% vs. SA-AET 97.08%), reflecting the deliberate trade-off against surrogate overfitting.

Relevance to Vision-Language Models¶

SimVLA directly probes the adversarial robustness of cross-modal alignment in VLPMs — a core property underlying contrastive pre-training objectives (CLIP, ALBEF, TCL). By showing that cross-modal word grounding is essential even during the attack design phase, the work surfaces a structural vulnerability: VLPMs' text encoders carry semantically irrelevant representations that inflate apparent robustness. The finding that complex loss functions (triplet, circle, multi-term contrastive) improve white-box but not black-box performance has direct implications for evaluating VLPM robustness claims in the literature. For researchers studying the geometry of joint text-image embedding spaces, the identification of "image-irrelevant words" via masking-based KL divergence offers a lightweight probe for cross-modal alignment quality.