Skip to content

Discrete Latent Features Ablate Adversarial Attack: A Robust Prompt Tuning Framework for VLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Discrete latent features resist adversarial attacks in VLM prompt tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DEFEAT is a robust prompt tuning framework for CLIP-based VLMs that discretizes latent image features via VQ-VAE to suppress adversarial perturbations, avoiding the high cost of full adversarial fine-tuning. By replacing continuous patch features with their nearest codebook entries, adversarial and clean examples are mapped to the same discrete codes, neutralizing the perturbation-induced distribution shift. On 15 datasets, DEFEAT achieves a +13.76% improvement in harmonic mean of robustness and accuracy over the prior SOTA at \(\epsilon = 4/255\).

Problem

Adversarially fine-tuning entire VLMs is computationally prohibitive at scale. Existing parameter-efficient adversarial prompt tuning methods (APT, FAP) still operate on continuous image features, which are highly susceptible to adversarial perturbations: even imperceptible \(\ell_\infty\) perturbations (\(\epsilon = 4/255\)) cause large distribution shifts in CLIP's latent space, dramatically degrading robustness. No prior work has exploited latent feature discretization as a defense mechanism in this setting.

Method

DEFEAT augments CLIP's frozen encoders with two trainable components: a learnable textual prompt and the PerturbShield module.

PerturbShield operates in two stages: 1. Grid-based discrete feature reconstruction: The ViT grid (patch) features \(I_\text{patch} \in \mathbb{R}^{N \times d_v}\) are passed through a VQ-VAE. Each patch vector is replaced by its nearest codebook entry \(e_k\): \(z_q^{(i)} = \arg\min_{e_k} \|z_e^{(i)} - e_k\|_2\), yielding \(\hat{I}_\text{patch}\). Using patch features (rather than the single CLS token) allows multiple codes to represent the image, preserving semantic content while neutralizing perturbations. 2. Semantic alignment projection: A learnable linear map \(W \in \mathbb{R}^{1 \times N}\) aggregates \(\hat{I}_\text{patch}\) into a single robust embedding \(\hat{I}_\text{proj} = W \cdot \hat{I}_\text{patch}\), regularized toward the original class feature via \(\mathcal{L}^\mathcal{I}_\text{reg} = \|\hat{I}_\text{proj} - I\|_1\).

Logits fusion averages predictions from the discrete branch (\(p^\text{vq}\), robust but lossy) and the original class token branch (\(p^s\), clean-accurate), trading off robustness and accuracy.

Prompt alignment regularization constrains learned prompts toward hand-crafted prompts: \(\mathcal{L}^\mathcal{T}_\text{reg} = \|t^s - t\|_1\), improving generalization and, empirically, robustness.

The full training loss is: $\(\mathcal{L}(x^a, t^s, t, y) = \mathcal{L}_\text{ce}(x^a, t^s, y) + \mathcal{L}_\text{VQ-VAE}(I_\text{patch}) + \lambda \mathcal{L}^\mathcal{I}_\text{reg} + \mu \mathcal{L}^\mathcal{T}_\text{reg}\)$ with \(\lambda=10\), \(\mu=20\). Adversarial examples are generated with PGD under the \(\ell_\infty\) threat model using prompts from the previous epoch.

Key Contributions

  • First use of latent feature discretization (VQ-VAE on ViT patch features) as an adversarial defense within VLMs.
  • PerturbShield module combining grid-based VQ reconstruction and semantic alignment projection, end-to-end trainable alongside prompt vectors.
  • Logits fusion strategy that explicitly balances clean accuracy and adversarial robustness without additional hyperparameter tuning per dataset.
  • Empirical demonstration that prompt alignment regularization (hand-crafted → learnable) also boosts adversarial robustness, not just generalization.
  • Evaluation across 15 datasets under three settings: adversarial few-shot classification, adversarial cross-dataset generalization, adversarial domain generalization.

Results

  • Adversarial few-shot classification (16-shot, \(\epsilon=4/255\), avg. 11 datasets): DEFEAT achieves H=42.23% vs. APT 29.01% and FAP 28.47%; robustness 36.05% vs. APT 20.26% / FAP 19.82%.
  • Harmonic mean improvement over SOTA (FAP): +13.76% at 16-shot, \(\epsilon=4/255\).
  • Robustness improvement over FAP: +16.23% at 16-shot, \(\epsilon=4/255\).
  • At \(\epsilon=1/255\), 16-shot: DEFEAT H=62.83%, robustness=60.77% vs. FAP H=56.30%, robustness=48.86%.
  • Gains are consistent across all shot counts (1, 4, 16) and both perturbation sizes (\(\epsilon \in \{1/255, 4/255\}\)).
  • Baselines: zero-shot CLIP, APT (textual prompt tuning), FAP (multimodal prompt tuning); all use ViT-B/32 with TeCoA-pretrained encoder.

Limitations

  • Evaluated only on ViT-B/32 CLIP; generalization to larger backbones (ViT-L, etc.) or non-CLIP VLMs is not demonstrated.
  • PerturbShield introduces additional VQ-VAE parameters and a second forward pass through patch features, adding inference cost not quantified in the paper.
  • Defense is validated against PGD (\(\ell_\infty\)); robustness to adaptive attacks that specifically target the discrete quantization step is not assessed.
  • Performance on very limited data (1-shot) at \(\epsilon=4/255\) shows only modest gains in accuracy, suggesting the VQ-VAE may overfit in extreme low-data regimes.
  • The codebook size \(K\) and its sensitivity are not fully ablated across diverse domain types.

Relevance to Vision-Language Models

DEFEAT directly addresses a core vulnerability of CLIP-family VLMs: the fragility of continuous visual representations under adversarial perturbation, which is increasingly critical as VLMs are deployed in safety-sensitive settings. The insight that VQ codebook quantization acts as a natural perturbation filter—mapping nearby adversarial and clean features to the same discrete code—offers a principled, architecture-agnostic defense principle applicable to any ViT-based VLM. By integrating this defense into the prompt tuning paradigm (CoOp/MaPLe lineage), DEFEAT shows that feature-space defenses and PEFT can be jointly optimized without touching encoder weights, a practically important finding for the broader VLM robustness community. This work extends the emerging line of adversarial prompt tuning (APT, FAP) by shifting focus from loss-function design to feature representation, opening a new axis for future robustness research.