Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Mechanistic defense against typographic attacks in CLIP; ICLR 2026 analysis

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dyslexify is a gradient-free defense against typographic attacks in CLIP that works by identifying and ablating a sparse "typographic circuit" of attention heads in the vision encoder's latter layers. Using a novel Typographic Attention Score to localize these heads, circuit ablation improves robustness by up to +22% on ImageNet-100-typo while degrading standard zero-shot accuracy by less than 1%, without any fine-tuning.

Problem

CLIP-based models are vulnerable to typographic attacks—embedding adversarial text in images causes misclassification and can jailbreak downstream VLMs. Existing defenses (fine-tuning, learnable projections, Defense-Prefix, Sparse Autoencoders) all require gradient-based optimization, making them computationally expensive, opaque, and hard to scale to billion-parameter models. No prior work identified the specific internal mechanisms responsible for this vulnerability.

Method

The method proceeds in three mechanistically motivated steps:

1. Localization via linear probes. Layer-wise linear probes on the CLS token of OpenCLIP models (ViT-B through ViT-BigG) trained on ImageNet-100 vs. ImageNet-100-typo show that typographic information (\(\text{Acc}(P_{\text{typo},\ell})\)) emerges abruptly in the second half of layers through attention blocks, while MLP blocks reduce linear decodability.

2. Typographic Attention Score (TAS). For each attention head \(H_{i,\ell}\), the score $\(T_{i,\ell} = \sum_{x \in D} \frac{\sum_{t=1}^{T} \mathbf{1}(t)\, A^*_{i,\ell,t}(x)}{\sum_{t=1}^{T} A^*_{i,\ell,t}(x)}\)$ measures the fraction of spatial CLS-attention concentrated on typographic image regions. Heads with \(T_{i,\ell} \geq \mu(T) + 2\sigma(T)\) are designated "typographic heads."

3. Circuit ablation. Heads are ranked by \(T_{i,\ell}\) and added to circuit \(C\) greedily: a head is included only if it improves typographic accuracy (\(\Delta\text{Acc}_\text{typo} > 0\)) and does not drop clean accuracy beyond tolerance \(\epsilon = 0.01\). Ablation zeroes only the head's contribution to the CLS residual stream (Eq. 5), leaving spatial tokens intact. The resulting circuits cover at most 10.1% of all attention heads.

Key Contributions

• Typographic Attention Score: a parameter-free metric that localizes attention heads causally responsible for transmitting typographic information to the CLS token.
• Dyslexify: a gradient-free circuit-ablation defense that requires no fine-tuning or learned parameters and scales to billion-parameter models on consumer hardware.
• Causal validation: manipulating CLS self-attention weight \(\alpha\) in circuit heads monotonically controls typographic susceptibility, providing direct causal evidence of the circuit's role.
• Medical use case: demonstrated defense of WhyLesionCLIP (skin lesion diagnosis on ISIC2019), where typographic attacks cause up to 22% accuracy drop and Dyslexify recovers up to 19.3%.
• Model release: family of "dyslexic" OpenCLIP variants (ViT-B/L/H/G/BigG) as drop-in replacements.

Results

• ImageNet-100-typo (synthetic): +19.90% (ViT-B), +20.32% (ViT-L), +21.26% (ViT-H), +22.06% (ViT-G), +16.74% (ViT-BigG) over baseline CLIP.
• Real Typographic Attack-100 (RTA-100): +11.90% to +16.60% across model sizes.
• Disentangling dataset: up to +31.11% (ViT-B).
• Non-typographic benchmarks (Aircraft, Food-101, ImageNet-100): deviations within ±1% in nearly all cases; worst case −1.74% on Aircraft for ViT-L.
• Vs. Defense-Prefix (ViT-L): Dyslexify outperforms on 2/3 typographic benchmarks (RTA-100: +16.60 vs. +7.80; PAINT: +14.55 vs. +10.01) while DP wins on Disentangling (+32.20 vs. +10.00).
• Medical (ISIC2019-typo): typographic attack degrades melanoma detection by up to 22%; Dyslexify recovers up to +19.3% and improves clean accuracy on 3/4 datasets.

Limitations

• Ablation targets only the CLS token residual; spatial tokens still carry typographic information, leaving VLMs that consume spatial tokens (e.g., LLaVA, IP-Adapter) potentially unprotected.
• Adaptive attacks cannot be evaluated because typographic attacks are non-differentiable, so robustness against adversaries aware of the circuit ablation is unquantified.
• Circuit construction requires a calibration dataset and two evaluation sets; the transferred robustness depends on how well these datasets represent real attack distributions.
• The identified mechanistic insights could be exploited offensively to craft inputs that increase spatial attention in circuit heads, amplifying attack effectiveness.

Relevance to Vision-Language Models

Typographic attacks are a first-class threat to VLMs: adversarial text in images can hijack goal-directed behavior, bypass safety filters, and cause jailbreaks in multimodal systems built on CLIP backbones. Dyslexify's mechanistic finding—that a sparse set of late-layer attention heads forms a causal "typographic circuit"—directly informs how CLIP-based vision encoders should be analyzed and hardened before integration into larger VLM stacks. The gradient-free, drop-in nature of the fix is especially relevant for practitioners deploying frozen CLIP encoders in production VLMs. However, the open challenge of protecting spatial tokens means Dyslexify is not yet a complete solution for architectures like LLaVA that pass the full spatial feature map to language decoders.