Skip to content

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

🕒 Published (v1): 2026-03-31 08:17 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; alignment-guided fine-tuning preserves zero-shot adversarial robustness in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AGFT is an adversarial fine-tuning framework for CLIP that replaces hard-label supervision with soft probabilistic targets from the pre-trained model, combined with temperature-scaled distribution calibration, to improve zero-shot adversarial robustness without destroying cross-modal alignment. It outperforms all prior adversarial fine-tuning baselines across 15 zero-shot benchmarks. The key insight is that label-based methods collapse the rich image–text similarity structure, while soft alignment targets preserve it.

Problem

Existing zero-shot adversarial robustness (ZSAR) methods (e.g., TeCoA, GLADIATOR) fine-tune the CLIP image encoder using hard cross-entropy labels. This treats CLIP as a conventional classifier, ignoring the relative similarity structure across all candidate texts. The result is degraded cross-modal alignment and reduced zero-shot generalization after adversarial fine-tuning—a robustness–generalization trade-off that hard labels exacerbate.

Method

AGFT has two components applied during adversarial fine-tuning of the CLIP image encoder on ImageNet (text encoder frozen):

Text-guided adversarial training. Instead of one-hot labels, AGFT uses the pre-trained model's own soft predictions as supervision targets: $\(p_{\text{orig}}^{i,j} = \frac{\exp\!\left(\cos(f_{\theta_{\text{orig}}}(x^i), f_\phi(t^j))/\tau\right)}{\sum_k \exp\!\left(\cos(f_{\theta_{\text{orig}}}(x^i), f_\phi(t^k))/\tau\right)}\)$ Adversarial examples are generated via PGD, and the loss minimizes KL divergence between the robust model's output and \(p_{\text{orig}}\).

Distribution consistency calibration. Directly targeting \(p_{\text{orig}}\) conflates the desired semantic similarity structure with the clean model's overconfident confidence scale. AGFT decouples these by temperature-scaling the pre-trained logits: $\(p_{\text{rob}}^{i,j} = \frac{\exp\!\left(\cos(f_{\theta_{\text{orig}}}(x^i), f_\phi(t^j))/(\tau/\gamma)\right)}{\sum_k \exp\!\left(\cos(f_{\theta_{\text{orig}}}(x^i), f_\phi(t^k))/(\tau/\gamma)\right)}\)$ where \(\gamma \in (0,1]\) softens the target distribution (set to \(\gamma=0.4\)), preserving cross-modal rank ordering while removing excess confidence that would mismatch the robust feature space.

The final objective is a standard min-max: $\(\min_\theta\, \mathbb{E}_{x \in \mathcal{D}}\!\left[\max_{x_{\text{adv}} \in B(x,\epsilon)} L(x_{\text{adv}}, \mathbf{t}, p_{\text{rob}}, \tau)\right]\)$

Key Contributions

  • AGFT framework: adversarial fine-tuning guided by soft cross-modal alignment targets rather than hard labels.
  • Distribution consistency calibration: temperature-scaling the pre-trained target distribution (\(\gamma=0.4\)) to decouple semantic structure from confidence scale during robust training.
  • Empirical demonstration that alignment-preserving supervision consistently outperforms label-based baselines across 15 datasets, 3 architectures, multiple perturbation budgets, and 15 diverse attack methods.

Results

  • Average zero-shot robust accuracy (PGD-20, \(\epsilon=1/255\), 15 datasets, ViT-B/32): AGFT 46.57% vs. GLADIATOR 43.46%, TGA-ZSR 40.84%, PMG-AFT 41.66%, TeCoA 38.51%.
  • Average zero-shot clean accuracy: AGFT 61.35% vs. GLADIATOR 60.34%, PMG-AFT 59.23%, TeCoA 56.93%.
  • Strong attacks (\(\epsilon=2/255\), PGD/C&W/AA): AGFT PGD 37.31%, C&W 33.79%, AA 33.34% — best across all three vs. GLADIATOR (30.56/29.37/28.49).
  • Large budget (\(\epsilon=4/255\)): AGFT clean 49.47% (+7.7%), robust PGD 24.98% (+3.8%) over best baseline.
  • ViT-B/16: AGFT robust 49.84% vs. PMG-AFT 47.08% (next best).
  • Diverse unseen attacks (15 attack settings): AGFT best average, outperforming strongest baseline by 3.6%.
  • Notable exception: PCAM robust accuracy — AGFT (33.89%) trails GLADIATOR (39.29%), showing one dataset where label-free alignment is not universally superior.

Limitations

  • Evaluated only on adversarial fine-tuning of CLIP; generalization to other VLM architectures (e.g., ALIGN, SigLIP, LLaVA) is not demonstrated.
  • Training remains on ImageNet only; the effect of using more diverse or domain-specific fine-tuning sets is unexplored.
  • The temperature scaling ratio \(\gamma\) requires manual tuning; sensitivity analysis is not shown in the excerpted text.
  • PCAM performance trails GLADIATOR, suggesting the soft-alignment objective may be suboptimal for certain distribution-shifted medical imaging tasks.
  • White-box threat model only; adaptive attacks that exploit the calibration mechanism are not evaluated.

Relevance to Vision-Language Models

AGFT directly targets a core tension in CLIP-based VLMs: adversarial robustness methods that improve \(\ell_p\)-norm resistance tend to erode the cross-modal embedding alignment that enables zero-shot generalization. By framing adversarial training as a distribution alignment problem rather than a classification problem, AGFT offers a principled recipe for hardening VLMs without sacrificing their foundational capability. This is directly relevant to anyone deploying CLIP or CLIP-derived models (e.g., in retrieval, grounding, or multimodal pipelines) in adversarially contested environments. The calibration mechanism also connects to broader work on knowledge distillation and soft-target training in VLMs, suggesting it could compose with other fine-tuning objectives (e.g., instruction tuning, domain adaptation).