Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model¶
🕒 Published (v1): 2026-07-21 10:49 UTC · Source: Arxiv · link
Why this paper was selected
ICT CAS (Shan/Chen/Gao); dual adversarial fine-tuning closes LVLM visual robustness gap
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
DAFT is a dual adversarial fine-tuning framework for CLIP's vision encoder that combines a visual supervision branch (L2 alignment between adversarial and clean features from a frozen reference encoder) with a semantic supervision branch (caption-guided triplet contrastive loss). The resulting encoder is a drop-in replacement for the CLIP ViT-L/14 backbone in LLaVA, conferring adversarial robustness across zero-shot classification, image captioning, and VQA without task-specific retraining.
Problem¶
Existing adversarial fine-tuning methods for LVLMs fall into two failure modes: (1) supervised methods (TeCoA, PMG-AFT) use ImageNet class-label embeddings, which overfit to classification and fail to generalize to captioning and VQA; (2) unsupervised methods (FARE) improve robustness transfer but remain vulnerable to sophisticated multimodal alignment attacks that target the image-text correspondence mechanism. No prior method achieves cross-task robustness spanning classification, captioning, and VQA simultaneously.
Method¶
DAFT fine-tunes only the CLIP image encoder (ViT-L/14) on COCO-caption with two complementary losses, then substitutes the trained encoder directly into LLaVA 1.5.
Visual supervision branch — prevents overfitting by pulling adversarial features toward clean reference features from a frozen copy of the original encoder: $\(\mathcal{L}_\text{vis} = \|F_\text{ori}(x) - F_\theta(x^a)\|_2^2\)$
Semantic supervision branch — aligns adversarial features to the corresponding ground-truth caption (positive \(Po\)) via triplet loss with the hardest-in-batch negative (\(Ne\), the non-matching caption with highest cosine similarity to the adversarial feature): $\(\mathcal{L}_\text{sem} = \max\!\left(\frac{F_\theta(x^a)\cdot T_\phi(Ne)}{\|F_\theta(x^a)\|\|T_\phi(Ne)\|} - \frac{F_\theta(x^a)\cdot T_\phi(Po)}{\|F_\theta(x^a)\|\|T_\phi(Po)\|} + a,\ 0\right)\)$
Combined objective and adversarial attack generation alternate via PGD: $\(\mathcal{L}_\text{DAFT} = \mathcal{L}_\text{sem} + \gamma \mathcal{L}_\text{vis}\)$
The text encoder \(T_\phi\) is frozen; only \(F_\theta\) is updated.
Key Contributions¶
- Dual supervision framework (visual + semantic) for CLIP adversarial fine-tuning that jointly addresses overfitting and multimodal alignment vulnerability.
- First use of descriptive captions (rather than class labels) as semantic supervision for adversarial training in LVLMs.
- Plug-and-play robustness transfer: a single fine-tuned encoder replaces the original CLIP backbone in LLaVA with no architecture modifications or per-task retraining.
Results¶
Evaluated under \(\ell_\infty\) PGD attack at \(\epsilon = 2/255\) and \(\epsilon = 4/255\); baselines are CLIP (no defense), TeCoA, PMG-AFT, and FARE.
Zero-shot classification (robust accuracy, \(\epsilon=2/255\), selected datasets): - CIFAR-10: DAFT 53.4% vs. FARE 49.9%, TeCoA 38.0%, PMG-AFT 37.9% - CIFAR-100: DAFT 25.8% vs. FARE 23.4%, TeCoA 12.4% - DTD: DAFT 24.7% vs. FARE 22.3%, TeCoA 11.3% - Caltech101: DAFT 69.8% vs. FARE 70.1% (FARE marginally better on this dataset)
At \(\epsilon=4/255\): - CIFAR-10: DAFT 28.1% vs. FARE 26.7% - StanfordCars: DAFT 7.1% vs. FARE 4.5%
- DAFT achieves the highest average robust accuracy across all 11 classification datasets at both \(\epsilon=2/255\) and \(\epsilon=4/255\).
- On LLaVA 1.5-7B VQA and captioning tasks under attack, DAFT produces correct answers where FARE and TeCoA fail qualitatively (Fig. 2); quantitative captioning (CIDEr) and VQA accuracy numbers are reported in the paper but fall outside the provided excerpt.
Limitations¶
- Fine-tuned exclusively on COCO-caption; smaller scale than ImageNet or LAION-400M may limit robustness coverage for fine-grained or domain-specific categories.
- Evaluation restricted to \(\ell_\infty\) PGD attacks; robustness against \(\ell_2\), patch, or physically realizable attacks is not assessed.
- Framework targets CLIP-backbone LVLMs (LLaVA); applicability to non-CLIP architectures (e.g., SigLIP-based models) is claimed but not empirically demonstrated.
- Triplet margin hyperparameter \(a\) and balancing coefficient \(\gamma\) require tuning.
Relevance to Vision-Language Models¶
Adversarial robustness of the shared CLIP vision encoder is a practical deployment bottleneck for a large class of LVLMs (LLaVA, MiniGPT-4, InternLM-XComposer), making encoder-level defenses with plug-and-play transfer highly impactful. DAFT advances the state of the art by showing that caption-guided contrastive supervision—already central to LVLM pre-training paradigms—is a stronger adversarial robustness signal than class-label supervision, directly bridging the VLM training methodology literature with the adversarial defense literature. The result that a single fine-tuned encoder generalizes across classification, captioning, and VQA reinforces the importance of multimodal alignment preservation (not just feature consistency) as a design criterion for robust VLM components.