Skip to content

Robustness in Both Domains: CLIP Needs a Robust Text Encoder

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP's text encoder has been entirely overlooked in adversarial robustness research despite its role in downstream multimodal systems. This paper introduces LEAF (Levenshtein Efficient Adversarial Finetuning), a character-level adversarial finetuning scheme for CLIP text encoders that mirrors the FARE approach used for image encoders. Combining LEAF with FARE yields the first CLIP model that is simultaneously robust in both the image and text input domains.

Problem

Prior work on making CLIP robust focused exclusively on the image encoder (TeCoA, FARE), leaving the text encoder vulnerable to character-level adversarial perturbations. This gap allows adversaries to degrade downstream tasks—text-to-image generation, multimodal retrieval, zero-shot classification—by perturbing textual inputs by as few as one character, without any defense on the text side.

Method

LEAF adapts the FARE objective to the text domain via a TextFARE loss: minimize the L2 distance between the original CLIP text encoder's embedding and the finetuned encoder's embedding under the worst-case Levenshtein-bounded character perturbation (distance k). The threat model constrains perturbations so no new English words appear (semantic constraint from Chanakya et al.), preserving semantics.

The key algorithmic contribution is a training-time attack that evaluates a constant ρ perturbations per sentence (versus O(|sentence| + nCharmer·|Γ|) for the Charmer baseline). It operates in two sequential forward passes over ρ×B sentences: (1) replace a whitespace in ρ random positions, pick the highest-loss position; (2) try ρ random characters at that position, pick the highest-loss character. This parallelizes across the batch. Encoders are trained independently (decoupled from image encoder), enabling drop-in replacement downstream without retraining pipelines.

Key Contributions

  • TextFARE objective: Levenshtein-constrained adversarial finetuning loss for text encoders, analogous to image-domain FARE.
  • LEAF attack: Constant-cost (ρ evaluations per sentence) training attack that is ~37–65× faster than Charmer-20 with negligible accuracy loss (Table 1: 3.23 s vs. 118.19 s per batch).
  • Semantic constraints during training: Shown to be critical for preserving image-domain performance when text robustness improves.
  • Bimodal robustness recipe: First demonstration that FARE (image) + LEAF (text) jointly yields a CLIP robust in both domains without joint training.
  • Embedding inversion benefit: Robust text encoders yield substantially better text reconstruction from embeddings (Word Recall: 34.4% → 46.4% for ViT-L/14; Token Recall: 38.9% → 52.0%).
  • Open-source models and code (github.com/LIONS-EPFL/LEAF, HuggingFace LEAF-CLIP).

Results

  • Zero-shot text classification (AG-News, k=1, ViT-L/14): adversarial accuracy 44.5% (CLIP) → 63.2% (LEAF+FARE); clean accuracy minimally affected (74.4% → 78.0%).
  • Zero-shot image classification (ImageNet, ε=2/255, ViT-L/14): robust image accuracy 47.6% (FARE only) → 46.0% (FARE+LEAF), negligible drop.
  • MS-COCO retrieval (k=2, Charmer-Con, ViT-L/14): Recall@1 30.66% (non-robust) → 40.22% (robust); ~10-point improvement on average across models.
  • Text-to-image (SDXL, CLIPScore): LEAF encoders consistently improve CLIPScore under adversarial noise at all k≥1 on MS-COCO validation.
  • Embedding inversion (MS-COCO, ViT-g/14): Word Recall 43.7% → 54.8%, Token Recall 48.1% → 60.6%, BLEU 5.6 → 12.2.
  • Generalization: Models trained at k=1 maintain highest adversarial accuracy at k=2,3,4,5 on AG-News.

Limitations

  • Image and text encoders are finetuned independently; joint training could yield further robustness gains at higher cost.
  • No robustness to token-level attacks (excluded due to semantic validity concerns).
  • Largest models (OpenCLIP-ViT-bigG, EVA-CLIP) not trained due to compute constraints.
  • Not evaluated on token-encoder tasks beyond zero-shot classification, retrieval, and generation (e.g., RAG).
  • Bimodal attacks (simultaneous image+text perturbation) are validated only at inference time, not trained against jointly.

Relevance to Vision-Language Models

CLIP is the dominant vision-language backbone in LMMs (e.g., Flamingo, LLaVA) and text-to-image pipelines (Stable Diffusion, SDXL, FLUX), making text-encoder robustness a direct concern for any system ingesting user text through CLIP. This work provides a drop-in robust text encoder requiring no retraining of downstream components, which is the same modularity property that made FARE adoption for image encoders practical. The embedding inversion finding also has implications for privacy and interpretability of VLM representations. Researchers tracking VLM safety, robustness, and multimodal retrieval should note this fills the previously unaddressed half of CLIP's attack surface.