Skip to content

Enhancing CLIP Robustness via Cross-Modality Alignment

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

COLA (CrOss-modaLity Alignment) is a training-free, test-time defense that improves CLIP's adversarial robustness by restoring image-text alignment via SVD-based subspace projection and optimal transport (OT). It addresses the core vulnerability that adversarial perturbations amplify the inherent modality gap in CLIP's feature space. Across 14 benchmarks, COLA yields an average +6.7% gain under PGD attacks without retraining.

Problem

CLIP's contrastive pretraining produces image and text features that naturally cluster in separate embedding regions (modality gap). Under adversarial perturbations (PGD, CW), image features scatter further from text prototypes, causing near-collapse in zero-shot classification accuracy. Prior defenses—adversarial fine-tuning, prompt tuning, test-time counterattacks (TTC)—either incur training cost, add inference latency, or do not explicitly correct this cross-modal misalignment.

Method

COLA operates at test time with two coupled components:

  1. Global feature alignment via subspace projection: All class text embeddings are stacked into a matrix Z ∈ ℝ^(d×KM), then SVD extracts the top-C principal components U_C. Each adversarial image embedding x̂ is projected onto the text-induced subspace: Π(x̂) = U_C U_C^⊀ x̂. This filters out adversarial noise components that lie orthogonal to the semantic subspace, provably reducing cosine similarity distortion (Δ^Π ≀ Δ).

  2. Local structural alignment via OT: The adversarial image and each class are modeled as discrete distributions over N=5 augmented views and M=50 LLM-generated text descriptions, respectively. Augmented views are weighted by prediction entropy (lower entropy → higher weight). The OT cost matrix uses projected features: C^Π_y(n,m) = 1 − cos(Π(xÌ‚âż), z^m_y). Classification picks the class minimizing the OT transport cost. The projection is embedded directly into the cost, which is shown to not increase transport distance and to yield larger decision margins (Îł(C^Π) > Îł(C)).

C=256 SVD components and the Sinkhorn algorithm are used for efficiency. The method is plug-and-play over any existing CLIP checkpoint.

Key Contributions

  • First training-free, architecture-free test-time defense specifically targeting modality misalignment in CLIP under adversarial attack.
  • SVD-based subspace projection that provably reduces cosine similarity distortion between adversarial and clean image features.
  • Unified OT framework that integrates subspace projection directly into the transport cost, with theoretical guarantees of larger decision margins.
  • Demonstrated compatibility as a plug-in module on top of adversarially fine-tuned CLIP variants (TeCoA, PMG, FARE) without any retraining.
  • Faster inference than TTC (28 min vs. 40 min on ImageNet, batch=128, ViT-B/32) with higher both clean and robust accuracy.

Results

  • ImageNet (PGD, Ï”=1/255): COLA 50.0% robust vs. TTC 40.0%, CLIP 1.1%; clean: 62.8% vs. 51.7% (TTC) vs. 62.1% (CLIP).
  • ImageNet + variants average (PGD): COLA 45.6% robust vs. TTC 24.4%, CLIP 5.0%; clean: 39.4% vs. 35.1% vs. 39.7%.
  • 9-dataset benchmark (PGD): COLA improves over TTC on all 9 datasets; e.g. Food +15.3%, SUN +5.5%, Eurosat +4.8% (absolute, robust).
  • CW attacks: +4.8% avg. over CLIP; e.g. ImageNet: 42.3% vs. 30.8% (TTC) vs. 1.1% (CLIP).
  • Large budget Ï”=4/255: Most baselines collapse to near 0%; COLA maintains meaningful accuracy (e.g. Caltech101 29.2% robust vs. TTC 24.6%).
  • Plug-in on fine-tuned models (PGD): On FARE, COLA raises robust accuracy by +10.8% over TTC-augmented FARE on 9-datasets; on TeCoA, +16.5% over baseline and +10.8% over TTC-augmented.
  • ViT-L/14: +35.8% robust accuracy over TTC on ImageNet (57.7% vs. 21.9%).
  • Ablation: OT with projected cost C^Π gives +1.3% over OT with unprojected cost C on 9-datasets robust (PGD: 46.2% vs. 44.9%).

Limitations

  • Inherits biases from the pretrained CLIP backbone; the text-induced subspace may encode dataset-specific linguistic/visual priors, limiting transfer to unseen domains.
  • Robustness under adaptive attacks (where the adversary knows and targets COLA's projection and OT steps) is not evaluated.
  • Fairness-aware robustness is not addressed—gains may not be uniform across demographic or semantic subgroups.
  • Computational cost scales with the number of augmented views (N) and LLM-generated descriptions (M=50 per class); at inference, OT solvers add latency proportional to class count K.

Relevance to Vision-Language Models

COLA directly addresses the modality gap—an intrinsic structural property of contrastive VLMs like CLIP—which becomes catastrophically amplified under adversarial conditions. For researchers tracking VLMs, this connects to a broader thread on the reliability of zero-shot CLIP classification in safety-critical settings (medical, autonomous driving), demonstrating that the alignment-at-training-time paradigm leaves a structural vulnerability exploitable at test time. The OT-based distribution-level alignment framework also generalizes naturally to other multi-modal architectures beyond CLIP, and the plug-in compatibility with fine-tuned variants (TeCoA, FARE) suggests a modular defense strategy applicable across the VLM fine-tuning ecosystem.