HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HQA-VLAttack is a transfer-based black-box adversarial attack on vision-language pre-trained (VLP) models that simultaneously perturbs text and image inputs without querying the victim model. It addresses a key flaw in prior methods—inadvertently reducing negative pair similarity alongside positive pair similarity—by using contrastive learning to explicitly repel positive pairs and attract negative pairs in feature space. It achieves state-of-the-art attack success rates on image-text retrieval, visual grounding, and image captioning tasks across both fused (ALBEF, TCL) and aligned (CLIP) VLP architectures.

Problem¶

Existing transfer-based adversarial attacks on VLP models (SGA, DRA) focus exclusively on reducing cosine similarity between matched (positive) image-text pairs. This inadvertently also reduces similarity of unmatched (negative) pairs, leaving the victim model still somewhat biased toward retrieving positive examples. Query-based black-box attacks (VLAttack) avoid this but incur prohibitive query costs. No prior transfer-based method jointly optimizes positive and negative pair distances.

Method¶

HQA-VLAttack operates in two sequential stages on a surrogate model:

Text Attack: For each word in the input text, a synonym set is built using counter-fitting word vectors (cosine similarity > τ = 0.4 in the counter-fitting embedding space), falling back to BERT-Attack MLM predictions when a word is absent from the vocabulary. The adversarial text is the single-word substitution that maximally reduces cosine similarity between the text and its paired image in the surrogate's embedding space (Eq. 4).

Image Attack (two sub-steps): 1. Layer-importance initialization: Layer importance weight w_{i,l} is set to the cosine similarity between the [CLS] token embedding at layer l and the final layer L_p (Eq. 5). This dampens over-weighting of top layers (which would cause surrogate overfitting) while still prioritizing high-influence layers. PGD minimizes the importance-weighted sum of per-layer, per-token cosine similarities between clean and adversarial images (Eq. 6). 2. Contrastive learning refinement: A contrastive loss (Eq. 7) simultaneously pushes adversarial image embeddings away from matched text embeddings (positive pairs, penalized by factor λ = −10) and pulls them toward unmatched text embeddings (negative pairs) across a batch (B = 16). Multi-scale image transformations (S ∈ {0.50, 0.75, 1.00, 1.25, 1.50}) and augmented caption sets improve transferability. PGD is used to optimize this objective.

Key Contributions¶

Identification of the negative-pair similarity problem in existing transfer-based VLP attacks, demonstrated empirically in Figure 1.
Layer-importance weighting scheme for initializing image adversarial examples that avoids surrogate model overfitting by using [CLS]-token similarity profiles rather than equal-weight or gradient-magnitude heuristics.
Contrastive learning loss that explicitly increases negative pair similarity while decreasing positive pair similarity, directly targeting the retrieval objective of VLP models.
Counter-fitting word vector substitute set generation that provides semantically consistent synonyms without relying solely on context-sensitive MLM predictions.
Demonstrated cross-task and cross-architecture transfer, including successful adversarial examples against closed-source MLLMs (GPT-4o, Claude 3.7 Sonnet).

Results¶

Flickr30K (same-architecture transfer, ALBEF→TCL): TR R@1 +23.28%, IR R@1 +18.77% over DRA (prior SOTA).
Flickr30K (same-architecture transfer, CLIPViT→CLIPCNN): TR R@1 +28.60%, IR R@1 +26.45% over DRA.
Flickr30K (cross-architecture, ALBEF→CLIPViT): TR R@1 +13.01%, IR R@1 +13.66% over DRA.
MSCOCO (ALBEF→TCL): TR R@1 +25.00%, IR R@1 +20.35% over DRA.
MSCOCO (CLIPViT→ALBEF): TR R@1 +20.09%, IR R@1 +18.03% over DRA.
White-box: Near 100% ASR on both TR and IR for CLIPViT on Flickr30K and MSCOCO.
Cross-task (ITR→IC, surrogate ALBEF, victim BLIP on MSCOCO): BLEU-4 reduced to 31.8 (vs. 34.8 for SGA/DRA); CIDEr reduced to 104.6 (vs. 115.9 for DRA), a −11.3 point gap.
Cross-task (ITR→VG, RefCOCO+ Val): Accuracy 46.48 vs. 53.88 for DRA.
Ablation: All four components (CF word vectors, layer importance, initial generation, contrastive optimization) each contribute positively; larger batch sizes monotonically improve ASR up to B = 16.

Limitations¶

Image perturbation bound is small (ε_v = 2/255), restricting attack strength; the trade-off between imperceptibility and ASR is not explored systematically.
Text perturbation is restricted to single-word replacement, which may be insufficient for longer or more complex inputs.
Counter-fitting vocabulary coverage is incomplete; BERT-Attack fallback reintroduces potential semantic inconsistency for out-of-vocabulary words.
Contrastive learning batch size B = 16 is a practical compromise; performance continues to scale with larger B but computational cost grows.
Cross-MLLM transfer experiments (GPT-4o, Claude 3.7 Sonnet) are qualitative/illustrative only; no quantitative ASR numbers are reported against MLLMs.
No defense evaluation is included; robustness of HQA-VLAttack against adversarial purification or certified defenses is unknown.

Relevance to Vision-Language Models¶

This paper directly probes a security vulnerability intrinsic to the joint embedding spaces that underlie most modern VLMs—the contrastive similarity objective used for retrieval can be inverted to cause systematic misretrieval. The insight that prior attacks inadvertently "help" negative pairs is a non-obvious structural observation about how cross-modal embeddings are organized, relevant to anyone studying the geometry of VLP feature spaces (CLIP, ALBEF, BLIP, etc.). For VLM researchers, the cross-task and cross-model transfer results (including to proprietary MLLMs) suggest that adversarial vulnerabilities are deeply shared across architectures, not model-specific, which has implications for robustness benchmarking and defense design. The contrastive learning formulation used for attack mirrors the training objective of many VLMs, highlighting a potential adversarial attack surface inherent to contrastive pre-training itself.