Skip to content

Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP

🕒 Published (v1): 2025-08-28 09:08 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dyslexify is a gradient-free defense against typographic attacks in CLIP that works by identifying and ablating a sparse set of attention heads ("typographic circuit") causally responsible for extracting text injected into images. It achieves up to +22% robustness improvement on typographic ImageNet-100 variants with <1% drop in standard zero-shot accuracy, and is validated on a medical CLIP-based skin lesion classifier.

Problem

CLIP-based models—including medical foundation models—are vulnerable to typographic attacks, where adversarial text overlaid on an image can cause misclassification (e.g., labeling a malignant lesion as benign). Prior defenses require gradient-based fine-tuning, are computationally expensive, and lack mechanistic interpretability. No gradient-free, interpretability-grounded defense existed before this work.

Method

The authors use linear probes across CLIP layers to localize typographic understanding, finding it emerges abruptly in the latter half of the vision encoder via attention blocks (not MLPs). They introduce the Typographic Attention Score (TAS): for each attention head, it measures how much the head's cls-token attention is directed toward spatial patches containing typographic content, using a dataset with synthetically localized text. Heads are ranked by TAS and greedily added to a typographic circuit C. Circuit ablation zeros out the contribution of these heads to the cls-token residual stream only (spatial tokens unchanged), subject to a tolerance ε = 1% accuracy drop on non-typographic data. The full algorithm is gradient-free and runs at inference time with no retraining.

Key Contributions

  • Mechanistic localization: Identifies a small subset of CLIP attention heads (≤10.1% of all heads) that causally transmit typographic content to the cls token via causal manipulation experiments.
  • Typographic Attention Score: A data-driven head-ranking metric guiding sparse circuit construction.
  • Dyslexify: Gradient-free, fine-tuning-free typographic defense applicable to models from ViT-B to ViT-BigG (billion parameters) on consumer hardware.
  • Medical validation: Demonstrates typographic attacks reduce melanoma detection accuracy by up to 22% on WhyLesionCLIP (OpenCLIP-ViT-L fine-tuned on ISIC2019); Dyslexify recovers up to +19.3% under attack and improves clean accuracy on 3/4 test sets.
  • Model release: Family of "dyslexic" OpenCLIP checkpoints as drop-in replacements for safety-critical pipelines.

Results

  • Typographic robustness (synthetic): Up to +22.06% on ImageNet-100-Typo (ViT-G); +28.68% on Food-101-Typo (ViT-H); +16.29% on Aircraft-Typo (ViT-BigG).
  • Typographic robustness (real-world): Up to +16.60% on RTA-100 (ViT-L); up to +31.11% on Disentangling dataset (ViT-B).
  • Clean accuracy cost: ≤1% drop on Aircraft, Food-101, ImageNet-100 in nearly all model sizes; largest single decline is −1.74% (ViT-L on Aircraft).
  • vs. Defense-Prefix (DP) baseline on ViT-L: Dyslexify outperforms DP on 2/3 typographic benchmarks (+16.60 vs. +7.80 on RTA-100; +14.55 vs. +10.01 on PAINT); DP leads on Disentangling (+32.20 vs. +10.00).
  • Medical (WhyLesionCLIP): Typographic attacks degrade zero-shot melanoma detection by up to 22%; Dyslexify recovers up to +19.3% under attack while improving clean accuracy on 3/4 ISIC datasets.

Limitations

  • Circuit ablation targets only the cls token; spatial tokens still carry typographic information, limiting effectiveness for VLMs (e.g., LLaVA, IP-Adapter) that use spatial features downstream.
  • Evaluation against adaptive attacks is not feasible because typographic attacks are non-differentiable, leaving open the question of robustness to attacks explicitly optimized against Dyslexify.
  • Circuit construction requires a small calibration dataset (typographic and clean), adding a setup step; hyperparameters ε and k must be chosen per model/domain.
  • Published results show one notable clean-accuracy outlier (ViT-L, −1.74% Aircraft), suggesting head selection is not perfectly clean-neutral for all fine-grained tasks.

Relevance to Foundation Models in Medicine

Medical CLIP-based foundation models (e.g., WhyLesionCLIP for skin lesions, MedCLIP, PubMedCLIP) inherit CLIP's vulnerability to typographic attacks, meaning adversarial text on a dermoscopy or radiology image can cause systematic misdiagnosis—a directly patient-harmful failure mode. Dyslexify provides a plug-in, no-retraining hardening step for any CLIP-derived medical model, making it immediately applicable to already-deployed systems without disrupting fine-tuning workflows. The mechanistic interpretability angle—mapping which attention heads encode clinically irrelevant typographic shortcuts—is valuable for auditing medical AI for safety-relevant spurious correlations. This work motivates routine adversarial robustness evaluation (including typographic attacks) as part of medical foundation model validation, especially for zero-shot or retrieval-based diagnostic systems.