Identifying Robust Neural Pathways: Few-Shot Adversarial Mask Tuning for Vision-Language Models¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Adversarial mask tuning identifies robust neural pathways in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AdvMask learns sparse binary masks over CLIP's frozen vision-encoder weights to identify "robust neural pathways" that resist adversarial perturbations in few-shot (1–16 shot) downstream settings. A companion Layer-wise Adaptive Feature Alignment (LAFA) loss aligns intermediate clean and adversarial features with confidence-weighted supervision, providing richer gradient signal than output-only objectives. AdvMask consistently outperforms adversarial prompt-tuning baselines across 11 classification benchmarks while storing only binary masks rather than modified weights.

Problem¶

CLIP and related VLMs collapse under adversarial attack (clean accuracy ~62% drops to ~2.5% under PGD), yet existing defenses have critical gaps: adversarial prompt tuning ignores the model's internal structure, direct fine-tuning overfits in few-shot regimes, and zero-shot robustness methods require large held-out datasets while still underperforming on task-specific data. No prior work frames the problem as finding an adversarially robust subnetwork within the pre-trained VLM.

Method¶

Binary mask tuning. A real-valued mask $M$ of the same dimensionality as the vision encoder weights $\theta$ is binarized via $M_{\text{bin}} = \mathbf{1}[M > \alpha]$. The masked encoder uses $\theta' = \theta \odot M_{\text{bin}}$ while $\theta$ stays frozen. Because binarization is non-differentiable, gradients flow through the Straight-Through Estimator: $M \leftarrow M - \gamma \cdot \partial\mathcal{L}/\partial M_{\text{bin}}$. Masks are applied only to multi-head self-attention layers (~20% of encoder parameters).

Adversarial training objective. Standard PGD min-max formulation: the inner step generates $\tilde{x} = \arg\max_{\|\delta\|_\infty \leq \epsilon} \mathcal{L}(f_{\theta \odot M_{\text{bin}}}(\tilde{x}, t), y)$ using 2-step PGD ($\epsilon = \alpha = 1/255$) during tuning; evaluation uses 100-step PGD.

LAFA loss. At every transformer layer $l$, the squared $\ell_2$ distance between the masked encoder's clean and adversarial intermediate features is minimized with per-sample confidence weighting: $$\mathcal{L}_{\text{LAFA}} = \frac{1}{|\mathcal{L}||\mathcal{B}|} \sum_{l \in \mathcal{L}} \sum_{x \in \mathcal{B}} \frac{p(y|x)}{\mathbb{E}_{x' \sim \mathcal{B}}[p(y'|x')] + \epsilon} \|z_{\text{clean}}^{(l)} - z_{\text{adv}}^{(l)}\|_2^2$$ High-confidence clean samples contribute stronger alignment targets; low-confidence ones are down-weighted to avoid misleading optimization under data scarcity.

Final objective: $\mathcal{L}_{\text{mask}} = \mathcal{L}_{\text{CE}}(\tilde{x}, y) + \lambda \cdot \mathcal{L}_{\text{LAFA}}(x, \tilde{x}, y)$, with $\lambda = 50.0$.

Key Contributions¶

Introduces the concept of robust neural pathways in pre-trained VLMs — adversarially stable subnetworks discovered via binary mask tuning without weight modification.
AdvMask framework: parameter-efficient adversarial tuning storing only binary masks over attention weights; compatible with existing prompt-tuning methods (e.g., CoOp).
LAFA loss: layer-wise, confidence-weighted intermediate-feature alignment that supplies dense supervision across transformer blocks rather than only at the output embedding.

Results¶

Zero-shot CLIP baseline: 61.9% clean / ~2.5% adversarial accuracy averaged over 11 datasets.
Few-shot adversarial robustness (Fig. 2, 11 datasets): AdvMask consistently exceeds all baselines (AdvVP, AdvVLP, AdvMaPLe, FAP) across 1–16-shot settings.
Base-to-new generalization (16-shot, Table 1):
AdvMask: 43.6% adv on base, 26.1% adv on new, harmonic mean H = 41.9
FAP (prior best): 38.0% / 21.9% / H = 37.6
Zero-shot transfer (trained on 16-shot TinyImageNet, tested on unseen datasets, Table 2):
AdvMask: 42.0% clean / 19.4% adv
FAP: 36.0% / 16.8%; TGA-ZSR (full dataset, ~31×more data): 38.6% / 22.9%
Clean accuracy recovers and can exceed original CLIP at 8–16 shots (e.g., Caltech101) despite no explicit clean loss, suggesting a regularization benefit from the learned mask.

Limitations¶

Adversarial evaluation is restricted to $\ell_\infty$ PGD; robustness to other threat models (e.g., $\ell_2$, AutoAttack) is not demonstrated in the main text.
Clean accuracy degrades in 1–2-shot regimes, similar to baselines, due to overfitting on scarce adversarial examples.
Masks are applied exclusively to multi-head self-attention layers; feed-forward and normalization layers are left unmasked — this design choice is inherited from prior mask-tuning work rather than ablated.
The text encoder is not masked; only the vision encoder is hardened, which may leave text-side vulnerabilities unaddressed.
Evaluated primarily on CLIP ViT-B/32; other backbones and VLMs (ViT-B/16, ViT-L/14, VisualBERT) appear only in the appendix.

Relevance to Vision-Language Models¶

This work directly targets a practical deployment gap for CLIP-style VLMs: their near-total collapse under adversarial inputs despite strong clean-sample performance. The parameter-efficient, weight-freezing design aligns with the field's trend toward adapter-based and mask-based tuning for large pre-trained models, and the LAFA loss advances the understanding of how intermediate representations in vision transformers propagate and amplify adversarial noise. The demonstrated transferability of learned masks to unseen datasets provides evidence that some adversarially vulnerable parameters in CLIP are globally problematic rather than task-specific — a finding with implications for model analysis and robust CLIP-based pipelines (retrieval, grounding, embodied agents). By outperforming prompt-based adversarial tuning with far fewer parameter updates, AdvMask offers a pragmatic path to hardening VLMs in low-data settings like medical imaging or satellite analysis.