ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToxicTextCLIP is a framework for text-based poisoning and backdoor attacks on CLIP during pre-training, addressing the underexplored vulnerability of the text modality. It uses a background-aware selector and a background-driven augmenter to generate semantically coherent adversarial texts that outperform prior baselines and bypass three state-of-the-art defenses.

Problem

Existing attacks on CLIP focus almost exclusively on the image modality. Text-based attacks are underexplored despite texts being more stable than images (not disrupted by compression/cropping), enabling persistent cross-platform propagation. Prior text-based approaches (e.g., naive caption substitution) suffer from two key flaws: (1) background content in target-class texts may be semantically inconsistent with the target class, weakening poisoning; and (2) many ImageNet classes lack enough high-quality, background-consistent texts in open corpora (>50% of classes cannot support even 30 poisoned texts per class at 0.003% of a 1M corpus).

Method

ToxicTextCLIP operates in two iterative modules:

Background-aware Target Text Selector: Given target-class texts, it enumerates all word-removal subsets up to η words to isolate background candidates. For each candidate background string, it scores it by the difference between its CLIP cosine similarity to the paired image and its similarity to a class centroid ZB (averaged over multiple prompt templates). Texts are ranked by how well their extracted background aligns with the class centroid, ensuring semantic coherence.

Background-driven Poisoned Text Augmenter: The top-ranked texts are encoded via the CLIP text encoder; their features are blended with corresponding image features (fjT ← fjT + λfjI) to inject visual context without semantic drift. A 6-layer Transformer decoder conditioned on both the blended text feature and image patch embeddings (via cross-attention) generates diverse candidate texts using Diverse Beam Search (DBS). A Jaccard-similarity-based post-processing step iteratively selects the least-redundant candidates, yielding a diverse final poisoned set.

For backdoor attacks, multiple target-class images are sampled; a trigger phrase (word-level: rare token "zx"; sentence-level: "Please return high-quality results.") is appended to the generated poisoned texts.

Key Contributions

• First systematic study of text-based poisoning and backdoor attacks during CLIP pre-training, identifying the text modality as a critical underexplored threat surface.
• ToxicTextCLIP framework with background-aware selector (ranking texts by background–class alignment) and background-driven augmenter (visual-textual decoder + DBS + Jaccard dedup).
• Empirical demonstration that ToxicTextCLIP bypasses RoCLIP, SafeCLIP, and CleanCLIP defenses across classification and retrieval tasks.
• Proposal of two defense directions: LM-based text anomaly detection and cross-modal background verification.

Results

• Poisoning (STI-P) on CC3M: ToxicTextCLIP achieves 95.83% ASR vs. 62.5% for mmPoison baseline; clean accuracy 32.23% vs. 31.52%.
• Word-level backdoor (W-BD) on CC3M: Hit@1 = 92.66% vs. 72.13% baseline (+20.34 pp); Hit@10 = 100%.
• Sentence-level backdoor (S-BD) on CC3M: Hit@1 = 98.68% vs. 64.41% baseline (+34.27 pp); Hit@10 = 100%.
• Against RoCLIP (CC3M, STI-P): 70.83% ASR vs. 33.33% mmPoison.
• Against CleanCLIP (CC3M, STI-P): 75.00% ASR vs. 45.83% mmPoison.
• Against RoCLIP (CC3M, S-BD): Hit@1 = 91.15% vs. 57.82% baseline.
• Text quality: Perplexity of generated texts = 408.89 vs. 755.27 for original web texts (lower is better).
• Ablation confirms both modules contribute: removing either selector or augmenter degrades ASR; full framework achieves 95.83% vs. 87.50% (w/o selector) and 83.33% (w/o augmenter) on CC3M STI-P.

Limitations

• Attacks are evaluated only against CLIP with ResNet-50 backbone; generalization to larger ViT-based CLIP models is not demonstrated.
• The background-aware selector requires paired image–text data from the corpus (not purely text-only), and enumerating all η-word removal subsets is combinatorially expensive for long texts.
• Performance degrades on YFCC15M relative to CC3M, attributed to higher data noise, and no analysis of the noise's specific interaction with the attack is given.
• Defense evaluation is limited to three existing methods; no adaptive or text-aware defenses are tested, which would be the natural adversarial response.
• The threat model assumes a black-box setting with no control over whether poisoned samples are crawled into the final dataset, introducing real-world deployment uncertainty not quantified.

Relevance to Vision-Language Models

This paper directly attacks the cross-modal alignment mechanism that is the foundation of contrastive VLMs like CLIP, demonstrating that the text encoder—not just the vision encoder—is a viable and under-defended attack surface. For researchers tracking VLMs, it highlights a systemic security gap in web-scale pretraining pipelines that underlies many downstream VLM applications (zero-shot classification, retrieval, VQA). The background-sensitive text generation technique also contributes a methodological insight about the role of background semantics in cross-modal representation learning, relevant beyond security to understanding what CLIP's contrastive objective actually captures. The paper motivates the development of modality-aware data curation and anomaly detection as necessary components of future robust VLM pretraining.