Skip to content

Towards Robustness against Typographic Attack with Training-free Concept Localization

🕒 Published (v1): 2026-07-02 17:55 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; training-free CLIP typographic attack defense — critical for all CLIP-encoder VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP vision encoders are vulnerable to typographic attacks (TA) — in-image text that biases classification toward lexical rather than visual semantics. This paper proposes a training-free mechanistic method that uses stochastic concept vector sampling inside ViT attention heads to identify lexical-focusing circuits, then suppresses them via attention reweighting or zero ablation. The approach outperforms supervised and prior training-free defenses across five ViT scales and multiple LVLMs.

Problem

CLIP-based vision encoders, which underpin LVLMs such as LLaVA, Qwen-VL, and InternVL, cannot decouple visual features from in-image text semantics. Existing defenses either require labeled training data and prompt tuning (Defense-Prefix) or rely on expensive greedy/parametric dictionary learning (Dyslexify). No prior method offers interpretable, low-cost, label-free localization of the precise ViT attention heads responsible for typographic vulnerability.

Method

The method operates in three stages:

  1. Stochastic concept mining ("Stochastic Lottery"). For each attention head in the last 20% of ViT blocks, \(K = 16 \times d_{\text{head}}\) random direction vectors \(\mathbf{u} \sim \mathcal{N}(0, \frac{1}{d_{\text{head}}}I)\) are sampled in the head's \(d_{\text{head}}\)-dimensional OV subspace. Operating in \(d_{\text{head}}\) rather than \(d_{\text{model}}\) reduces polysemantic interference: the maximum interference is bounded as \(\max_i |I_{\mathbf{u},i}| \le \|\boldsymbol{\xi}_{\text{max}}\|\sqrt{\frac{4\log N}{d_{\text{head}}}}\), making it feasible to recover concept-faithful probes without parametric training.

  2. Gradient-based lexical attribution. For each sampled \(\mathbf{u}\), the normalized Text Attribution Score (nTAS) is computed via \(\frac{\partial F_i(\mathbf{u})}{\partial s_{i,j}} = A_{i,j}(V_j(\mathbf{u}) - F_i(\mathbf{u}))\), which jointly weighs routing gate strength and concept-patch alignment. The MAS/nTAS scores (Eqs. 13–14) contrast attribution to text-region patches vs. non-text patches; heads are flagged as lexical circuits via z-test at threshold \(z=1\).

  3. Mechanistic intervention. For object classification (CLIP zero-shot): attention reweighting of the <cls> token toward non-text patches (\(a=1\) in Eq. 12). For LVLM VQA (no <cls> token): zero ablation of the identified head's output. Circuit extraction requires only 1,280 unlabeled images and runs in under one minute per backbone on a single A100 (7.5 s for ViT-B/16, 45.7 s for ViT-bigG/14); test-time overhead is near-zero.

Key Contributions

  • Training-free stochastic concept mining in MHSA head subspaces, with formal analysis bounding polysemantic interference via concentration of measure.
  • Gradient-based attribution metric (nTAS) combining attention routing and concept alignment for label-free lexical circuit identification.
  • Mechanistic interventions (attention reweighting and zero ablation) applied to the extracted circuits, covering both CLIP classification and LVLM VQA settings.
  • New IN-100-Text benchmark with seven visual text styles generated via Qwen-ImageEdit for evaluating TA robustness on ImageNet-100.
  • Demonstrated generalization: improvements propagate to LVLMs (evaluated on RIO-Bench) without any LVLM-level fine-tuning.

Results

  • ViT-B/16 on RTA-100: OCA \(56.3 \to 68.7\) (+12.4 pp), TCR \(30.8 \to 12.6\) (−18.2 pp).
  • ViT-H/14 on RTA-100: OCA \(53.4 \to 76.2\) (+22.8 pp), TCR \(42.0 \to 14.4\) (−27.6 pp).
  • ViT-H/14 on Disentangling: OCA \(46.1 \to 82.2\) (+36.1 pp), TCR \(53.3 \to 17.2\) (−36.1 pp).
  • ViT-bigG/14 on PAINT: OCA \(49.5 \to 79.6\) (+30.1 pp), TCR \(38.8 \to 10.7\) (−28.1 pp).
  • Consistent gains across all five ViT scales (B/16, L/14, H/14, g/14, bigG/14) on RTA-100, Disentangling, PAINT, and IN-100-Text.
  • Outperforms both Defense-Prefix (supervised) and Dyslexify (training-free) on object classification benchmarks.
  • Applying zero ablation to LVLM vision encoders yields substantial VQA accuracy gains on RIO-Bench (easy/medium/hard subsets); full numbers truncated in provided text.
  • Clean ImageNet-100 accuracy is preserved (e.g., 83.2–83.9% across z-threshold choices for ViT-H/14).

Limitations

  • Intervention efficacy for VQA (zero ablation) is evaluated only on RIO-Bench; generalization to other LVLM benchmarks is not shown.
  • Circuit extraction requires knowledge of text patch locations (mask \(M\)) during the attribution phase; fully blind deployment (unknown text location) is not analyzed.
  • The method is validated on CLIP ViTs pretrained on LAION-2B; transferability to other CLIP training regimes or non-ViT vision encoders is unexplored.
  • z-score threshold selection uses a small calibration dataset; performance degrades at \(z=2\) (too few heads selected), indicating some sensitivity to this hyperparameter.
  • The new IN-100-Text dataset uses synthetic AI-generated text overlays; real-world typographic attack distributions may differ.

Relevance to Vision-Language Models

This work directly addresses the fragility of CLIP-based vision encoders that are the backbone of modern LVLMs, showing that TA vulnerability is mechanistically localized to a sparse set of attention heads rather than diffusely distributed. The training-free, interpretable intervention is practically significant because it applies to already-deployed LVLMs (LLaVA, Qwen-VL, InternVL) at near-zero cost, without any LVLM-level retraining. For researchers studying VLM robustness and safety, this paper provides both a diagnostic tool (nTAS circuit mining) and an actionable defense that transfers across model scales. It also exemplifies how mechanistic interpretability methods developed in the language model literature can be adapted to explain and fix cross-modal failure modes in vision-language systems.