Learning Robust Vision-Language Models from Natural Latent Spaces

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoAPT is an adversarial prompt tuning framework for CLIP that suppresses adversarial perturbation space via adaptive total variation regularization, then restores corrupted natural features by aligning with frozen natural CLIP representations in latent space. It achieves a three-way balance between natural generalization, adversarial robustness, and task-specific adaptation across 15 datasets and four benchmark types, without benchmark-specific hyperparameter tuning.

Problem

Existing adversarial prompt tuning methods for VLMs (e.g., FAP, AdvMaPLe) overfit to in-distribution adversarial samples, degrading natural generalization on OOD or unseen tasks. Full fine-tuning corrupts pre-trained feature distributions, creating a robustness–generalization trade-off. Masked image modeling (MIM)-inspired approaches inadvertently direct models toward high-frequency components where adversarial perturbations concentrate, worsening robustness.

Method

CoAPT maintains a frozen "natural CLIP" alongside a "robust CLIP" with trainable deep multimodal prompts (depth 9, length 4, both visual and textual branches) and LoRA-style low-rank residual modules (rank r ≪ d).

High-frequency suppression (Adaptive-FGP): Input images (natural or adversarial) are smoothed via an improved fast gradient projection TV algorithm. A spatially adaptive regularization map γ(v) = µ_base · (1 + µ_gain · Φ(v)) (Sobel edge response) reduces regularization at edges and increases it in flat regions, preserving structure while suppressing perturbations. Nesterov acceleration with cosine-similarity-based adaptive restart (restart when ∠(σ_k, σ_{k-1}) > 90°) achieves real-time convergence.

Latent-space reconstruction: The robust CLIP processes HF-suppressed images; its visual and text embeddings are aligned to the natural CLIP embeddings via a Gaussian RBF reconstruction loss (L_recon), recovering corrupted generalization features without pixel-space reconstruction.

Rényi regularization: α-order Rényi divergence (α = 2.5) between natural and adversarial similarity distributions (L_rényi) penalizes distributional discrepancy. Classification uses Rényi cross-entropy (L_rce). Total loss: L_coapt = κ₁L_recon + κ₂L_rényi + κ₃L_rce (κ₁=8, κ₂=1, κ₃=1). LLM-generated text augmentations enrich the language branch. Only prompts ϕ and low-rank modules φ are updated; CLIP backbone is frozen.

Key Contributions

• Novel adversarial prompt tuning paradigm that learns robust VLM representations guided by the latent space of a frozen natural CLIP, replacing pixel-level reconstruction with latent-level feature restoration.
• Real-time Adaptive-FGP: spatially adaptive TV regularization with Nesterov acceleration and cosine-similarity-based adaptive restart for efficient, edge-preserving high-frequency suppression.
• Rényi divergence regularization (α = 2.5) to minimize natural-vs-adversarial distributional discrepancy, with corresponding Rényi cross-entropy for classification.
• Single hyperparameter configuration across all 15 datasets and four benchmarks (no dataset-specific tuning).

Results

All comparisons are against FAP (few-shot adversarial prompt tuning, NeurIPS 2024) and vanilla CLIP on ViT-B/32.

• Few-shot (average over 11 datasets): CoAPT achieves consistently superior natural and robust accuracy across 1/2/4/8/16 shots; on 6 datasets robust accuracy exceeds baseline natural accuracy; on 5 datasets (e.g., DTD) robust accuracy exceeds Vanilla CLIP natural accuracy.
• Base-to-novel generalization (16-shot): Average HM of robustness +32.39% vs. FAP; average natural HM +13.09%; base natural accuracy +7.95%, novel natural accuracy +15.77%; robust HM on OxfordPets reaches 51.57%.
• Zero-shot: ImageNet natural 63.42% (+1.32% vs. FAP); ImageNet robust 51.18% (+29.58%); 10-target-dataset average natural 54.06% (−7.83% vs. FAP); average robust 43.90% (+20.01%).
• OOD generalization (vs. FAP): Average natural +8.81%, average robust +17.37%; ImageNet-A robust 9.72% (+8.52%), ImageNet-R robust 50.71% (+22.51%), ImageNet-Sketch robust 29.13% (+12.83%), ImageNetV2 robust 42.23% (+25.63%).
• Overall claimed improvement (across 15 datasets): +9.83% natural, +24.16% adversarial over SOTA.
• Ablation: Removing Adaptive-FGP causes robust accuracy to drop by ~31% in few-shot-16 and ~26% in base-to-novel; removing reconstruction loss similarly collapses robustness; removing Rényi reduces all metrics moderately.

Limitations

• Evaluated only on CLIP (ViT-B/32); generalization to other VLM architectures (e.g., ALBEF, larger ViT variants) is not demonstrated.
• Adversarial training uses only PGD (ℓ∞, ε=1/255); robustness against other attack types (C&W, AutoAttack at larger ε, black-box transferable attacks) is not reported.
• Adaptive-FGP adds per-image iterative optimization at training time (max 30 iterations); inference-time cost with HF suppression is not benchmarked.
• Low-rank residual module is acknowledged as sensitive to dataset-specific hyperparameters and not fully tuned, limiting its contribution on some datasets.
• Zero-shot natural accuracy on the 10-target average is slightly below FAP (−7.83%), indicating a residual natural-robustness trade-off is not fully eliminated.

Relevance to Vision-Language Models

CoAPT directly addresses a core vulnerability of foundation-model VLMs like CLIP: their susceptibility to adversarial perturbations without sacrificing zero-shot generalization, which is central to their utility. By keeping the pre-trained backbone frozen and training only lightweight prompts plus low-rank modules, the method aligns with the broader parameter-efficient adaptation paradigm (prompt tuning, LoRA) that dominates VLM fine-tuning research. The latent-space reconstruction objective — aligning robust representations to clean natural representations — offers a principled alternative to adversarial training that avoids catastrophic forgetting, a persistent challenge in VLM adaptation. The use of Rényi divergence to explicitly minimize natural-vs-adversarial distributional gap is a technique applicable beyond CLIP to any contrastive or similarity-based VLM.