Skip to content

Inducing Dyslexia in Vision Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Mechanistic VLM study reveals VWFA analog in CLIP vision encoders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper uses large-scale VLMs as computational proxies for the human brain to simulate dyslexia by functionally localizing and ablating visual-word-form-selective (VWF-selective) units. Ablating ~6.89% of MLP units in the language decoder of Qwen2-VL-72B selectively degrades reading performance below a clinical dyslexia threshold while preserving visual IQ and reasoning. The resulting model replicates key human dyslexic phenotypes: phonological deficits, intact orthographic processing, and font sensitivity matching clinical recommendations.

Problem

Dyslexia research relies on behavioral and neuroimaging methods that cannot directly test causal hypotheses about which neural substrates drive reading impairments. Specifically, the causal role of the Visual Word Form Area (VWFA) in dyslexia remains debated—it is unclear whether VWFA hypoactivation causes reading deficits or is a downstream consequence. No prior work has modeled brain disorders using system-level neural network models with controlled lesion experiments.

Method

  1. Functional localization: Using a neuroscience fMRI localizer paradigm (Saygin et al., 2016), the model (Qwen2-VL-72B) is presented with four stimulus categories—written words, scrambled words, face line-drawings, and object line-drawings. For each unit, a \(t\)-statistic is computed comparing word-image responses against the three control categories. Units are ranked by descending \(t\)-statistic; the top \(k\%\) constitute the VWF-selective set.

  2. Minimal subnetwork identification: VWF-selective units are ablated (activations set to zero) in the MLP gate-projection layers (model.layers.{i}.mlp.gate_proj) across all 80 transformer blocks of the language decoder. Mask size \(k\) is increased monotonically until ROAR accuracy drops below 65% (the clinical dyslexia threshold, defined as 1 SD below population mean). The minimal \(k\) satisfying this criterion is selected (~6.89%).

  3. Evaluation: The ablated model is benchmarked on ROAR (lexical decision: real word vs. pseudo word), Raven's Progressive Matrices (nonverbal visual IQ), Kempler Sentence Comprehension (syntactic VQA), and a phonology/orthography dissociation battery (homophones, pseudo-homophones, transposed-letter stimuli; Luke et al., 2023).

Key Contributions

  • First demonstration that VWF-selective units can be functionally localized in a VLM using a neuroscience localizer protocol, and that these units predict human fMRI VWFA responses.
  • Causal evidence (via ablation) that VWF-selective units are specifically necessary for reading, not general vision or language: ablating ~6.89% of MLP units drops ROAR by −32% (normalized) while leaving RAVEN unaffected (\(p \approx 0.75\)).
  • Computational replication of the human phonological deficit profile: ablated model impairs on phonologically confusable stimuli but shows no significant impairment on orthographically confusable stimuli.
  • Replication of font-sensitivity observed in dyslexic humans: the ablated model shows significantly better accuracy on clinically recommended fonts (OpenDyslexic, Comic Sans, KG Primary Penmanship) and worse on fonts difficult for dyslexics (Papyrus), while the intact model is font-invariant.
  • Generalization across VLM architectures: reading-specific deficits reproduced in Molmo-72B and PixTral-12B.
  • Establishes a general computational framework for modeling brain disorders via hypothesis-driven unit perturbation in state-of-the-art neural networks.

Results

  • ROAR (reading): VWF ablation → −32% normalized accuracy, falling below the 65% dyslexia threshold (\(p \ll 0.01\)); random unit ablation from the same layers does not drop below threshold.
  • RAVEN (visual IQ): Performance unchanged after VWF ablation (\(p \approx 0.75\)); random ablation significantly impairs RAVEN.
  • Kempler (sentence comprehension): Intact or slightly enhanced after VWF ablation.
  • Phonology: Ablated model accuracy decreases on homophones/pseudo-homophones, mirroring dyslexic human phonological deficits.
  • Orthography: Ablated model shows no significant impairment on transposed-letter stimuli, diverging from some dyslexic human profiles.
  • Font sensitivity: Significant accuracy improvement (\(p \ll 0.001\)) on OpenDyslexic, Comic Sans, KG Primary Penmanship, and Orange Arial; significant degradation on Papyrus (\(p \ll 0.01\)); intact model shows no font-dependent differences.
  • Neural alignment: Top-\(k\%\) VWF-selective units show significantly higher noise-normalized fMRI VWFA alignment than random unit subsets of equal size.
  • Layer specificity: Only MLP layers in the language decoder yield reading-selective ablation effects; vision encoder, visual merger, and self-attention ablations non-selectively impair both ROAR and RAVEN.

Limitations

  • The orthographic deficit component does not fully replicate human dyslexic profiles—the ablated model is unimpaired on transposed-letter stimuli, whereas many dyslexic humans show orthographic difficulties.
  • The dyslexia threshold (65% ROAR) is a heuristic operationalization; clinical dyslexia is a spectrum diagnosed by multiple criteria.
  • The framework abstracts away genetic, developmental, and environmental factors that contribute to dyslexia in humans—only the VWFA-hypoactivation hypothesis is modeled.
  • Ablation (setting activations to zero) is a coarse lesion method; real VWFA hypoactivation involves graded, distributed reductions in neural gain.
  • Results are primarily reported on Qwen2-VL-72B; it is unclear whether the exact localization and deficit profile generalizes to smaller or differently architected VLMs.
  • Response-time-based reading metrics (central to clinical ROAR) are not measurable in autoregressive inference; only accuracy is evaluated.

Relevance to Vision-Language Models

This work provides a novel use case for VLMs as mechanistic models of human cognition, demonstrating that VLMs are structured enough to host functionally specialized subnetworks that align with specific human brain regions. For the VLM research community, it validates the MLP layers of the language decoder as the locus of word-form knowledge—complementing mechanistic interpretability work on factual knowledge storage in MLP layers—and shows that targeted ablation can dissociate modality-specific capacities (reading vs. visual reasoning) within a single multimodal model. More broadly, the functional localization pipeline (contrast-based \(t\)-statistic unit ranking + minimal mask search) is a transferable methodology for probing and editing VLM representations in other domains.