Unveiling the Visual Counting Bottleneck in Vision-Language Models¶

🕒 Published (v1): 2026-05-28 16:20 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

Mrinmaya Sachan (ETH); ICML 2026; reveals visual counting extrapolation bottleneck

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMs collapse catastrophically on visual counting outside their training distribution, not because they cannot see the objects, but because they fail to map valid visual magnitudes to symbolic tokens. Using synthetic Go boards and linear probing, the authors isolate the failure to the final symbolic-mapping stage and explain it via a fractured magnitude hypothesis: VLMs learn disjoint, modality-specific statistical manifolds rather than a unified number space. Scaling data alone is insufficient; inductive priors for unified representations are needed.

Problem¶

VLMs generalize well within their training distribution but fail to enumerate object quantities exceeding those seen during visual training, even when they possess the correct symbolic labels (learned via text pretraining). Prior work lacked a rigorous mechanistic decomposition of where in the perception–reasoning–generation pipeline this extrapolation failure originates.

Method¶

The authors construct a two-part experimental framework:

Synthetic Toy VLM (Study 1): A 2-layer ViT encoder + 2-layer causal Transformer decoder trained on 19×19 Go boards with a decoupled curriculum: text pretraining on counting up to $N=99$, then visual alignment restricted to $N \in [0,49]$. This creates a visual extrapolation regime (VE: $50$–$99$, labels known but visual density unseen) and a full extrapolation regime (FE: $100$–$120$, both unseen). White-stone distractors with cardinality $\pm 30$ prevent global-density shortcuts; stone size (14×14 px) aligns with patch tokens.
Foundation model validation (Study 2): Qwen3-VL-32B-Instruct on a 6×6 Go board variant ($N \in [0,20]$, 32×32 stones matching the model's $2\times 2$ merged patch tokens), using zero-shot chain-of-thought prompting.

Diagnostic: Hidden Number Probing. A linear probe $f_\text{probe}: \mathbb{R}^d \to \{0,1\}$ is trained on frozen visual encoder embeddings from ID data only and applied at test time to recover the hidden number $N_H = \sum_{i=1}^L f_\text{probe}(z_i)$. Two gap metrics are derived: $$\text{Vision Gap} = |N_H - N_G|, \quad \text{Language Gap} = |N_P - N_H|$$

Three hypotheses are tested sequentially: (A) visual individuation fails, (B) magnitude awareness degrades in the language module, (C) symbolic mapping is broken.

Key Contributions¶

Decomposition of visual counting into three cognitive stages (individuation, magnitude awareness, symbolic mapping) and systematic experimental falsification of the first two as failure causes.
Linear probe methodology (hidden number probing) to extract and quantify visual quantity information in encoder embeddings and compare it against generated output.
Causal verification via token-masking: masking $k$ stone-associated latent tokens shifts the model's prediction by exactly $-k$.
Error topology analysis showing prediction collapse to attractor tokens ($49$, $90$, $99$, $9$) rather than Gaussian noise around ground truth.
Circuit-level evidence: attention-head activation sets for visual vs. textual counting are non-overlapping in 95.7% of $N=6400$ examples, mechanistically grounding the fractured magnitude hypothesis.
Validation that the same failure pattern persists in Qwen3-VL-32B despite trillion-token pretraining.

Results¶

Toy VLM – accuracy: Text counting reaches 100% up to $N=99$ (full VE regime); visual counting achieves 100% within ID ($N \le 49$) but collapses to 0% immediately at $N=50$, with a single anomalous spike at $N=99$.
Toy VLM – gap metrics: Vision Gap $\approx 0$ across all regimes (ID, VE, FE); Language Gap explodes at $N=50$ and remains large throughout VE and FE.
Comparative counting (magnitude awareness): Model achieves $>90\%$ accuracy on binary cross-modal cardinality matching in the VE regime ($N\in[50,99]$), despite 0% explicit enumeration accuracy there.
Circuit overlap: Critical attention-head sets for visual vs. textual counting are disjoint in 95.7% of 6400 test cases.
Qwen3-VL-32B: Visual counting collapses after the subitizing range ($N>5$), approaching 0% by $N=15$; Vision Gap remains $<0.1$; Language Gap diverges rapidly, replicating the synthetic lab findings at scale.

Limitations¶

Synthetic Go board stimuli are highly controlled; transfer of exact bottleneck localization to natural-image counting (occlusion, scale variation, semantic ambiguity) is not directly demonstrated.
Linear probing confirms linear separability of quantity in encoder embeddings, but does not rule out non-linear quantity degradation that a linear probe would miss.
Qwen3-VL evaluation uses a smaller grid ($6\times6$, max $N=20$) due to model capacity constraints, limiting the extrapolation range observable.
The study diagnoses the failure but does not propose or evaluate a concrete architectural fix enforcing unified magnitude representations.
Only one foundation model (Qwen3-VL-32B) is validated; breadth across model families remains untested.

Relevance to Vision-Language Models¶

This work delivers a mechanistic diagnosis of a well-known but poorly understood VLM failure mode—counting extrapolation—directly relevant to any researcher studying VLM compositional or systematic generalization. By ruling out perceptual and numerosity failures and pinpointing the symbolic-mapping stage as the locus of collapse, it reframes the problem from a data-scaling challenge to an architectural-alignment challenge, with direct implications for how visual and textual representation spaces should be jointly trained. The fractured magnitude hypothesis offers a generalizable lens for other cross-modal grounding failures (spatial reasoning, ordinal relationships) beyond counting. The hidden-number probing technique is a broadly reusable diagnostic tool for dissecting where modality-specific information is lost in any VLM pipeline.