The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs¶

🕒 Published (v1): 2026-07-10 15:50 UTC · Source: Arxiv · link

Why this paper was selected

Diagnoses counting failures as representation-verbalization gap; mechanistic insight actionable for VLM evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMs frequently encode the correct object count in their intermediate activations yet verbalize a wrong answer — a representational misalignment rather than a missing-knowledge failure. The authors demonstrate this with multi-probe analysis and SVCCA, then exploit the finding to build an inference-time error detector that selectively re-prompts the model, gaining up to 15.6 absolute percentage points in counting accuracy with no parameter updates.

Problem¶

Prior work documents that VLMs are poor at counting but treats this as a behavioral phenomenon. The fundamental open question is whether the correct count is (a) absent from internal representations, (b) present but misaligned with the generation pathway, or (c) encoded but corrupted during decoding. Without this mechanistic understanding, remedies remain ad hoc.

Method¶

The paper trains three families of linear and nonlinear probes — ground-truth count probe, output-count probe, and binary error detector — on the hidden states $x_l \in \mathbb{R}^d$ extracted from every layer $l$ of four VLMs (InternVL2-1B/4B, Qwen3-VL-2B/8B). Probe architectures range from a circular probe (encodes count as an angle in a learned 2D plane: $\hat{y} = \frac{10}{2\pi}\text{atan2}(w_1^\top x_l, w_2^\top x_l)$) to logistic regression and single-hidden-layer MLP.

Subspace alignment is measured by applying SVCCA to the weight matrices $W_{\text{gt}}, W_{\text{out}} \in \mathbb{R}^{c \times d}$ of co-trained probe pairs: extract top-$k$ left singular vectors, compute canonical correlations $\rho_1,\ldots,\rho_k$, and summarize as $\text{SVCCA} = \frac{1}{k}\sum_{i=1}^k \rho_i$.

Causal steering injects a perturbation at layer $l^*$ along the normalized probe-weight difference direction: $$d = \frac{w_y - w_{\hat{y}}}{\|w_y - w_{\hat{y}}\|_2}, \quad \tilde{x}_{l^*} = x_{l^*} + \alpha \cdot \text{RMS}(x_{l^*}) \cdot d$$ with $\alpha$ swept from 5 to 65.

Detector-guided self-correction: an error detector scores each first-pass activation $s = D^{(l^*)}(x_{l^*})$; when $s \geq \tau$, the model is re-prompted with a correction prompt asking it to reconsider, otherwise the first answer is kept.

Key Contributions¶

Construction of four controlled synthetic counting datasets (varying color/shape diversity, 1170 images each, 9 count classes) plus CountBench evaluation.
Extension of the Sun et al. (2025) multi-probe framework from LLM arithmetic to VLM counting, applied to both vision-encoder and text-decoder representations.
Novel application of SVCCA to probe weight matrices (not activations) to compare supervision-specific readout subspaces; MLP and Circular probes show near-orthogonal alignment (SVCCA $\approx 0.098$–$0.100$), confirming the correct count occupies a different subspace from the output.
Causal validation via activation steering: probe-derived directions improve accuracy while random directions degrade it.
Detector-guided self-correction achieving up to +15.6 absolute percentage-point improvement without fine-tuning.

Results¶

Nonlinear probes (logistic, MLP) exceed 80% F1 at recovering ground-truth counts from hidden states across all models; Qwen3-VL-8B approaches near-perfect scores.
Ground-truth probes plateau at earlier layers; output probes peak near final layers — the model encodes the correct count before committing to a verbalized answer.
Error detectors on InternVL2 activations achieve near-perfect F1; Qwen3-VL detectors score lower (fewer training errors available).
Mean weight-SVCCA: Linear = 0.623, Logistic = 0.320, MLP = 0.100, Circular = 0.098 — nonlinear probes reveal near-orthogonal subspaces.
Causal steering on InternVL2-1B: accuracy on diff_col_diff_shape rises from ~27% to ~46% (+19 pp); random direction control degrades performance monotonically.
Detector-guided self-correction vs. "Always Reprompt" and "Random-K" baselines: up to +15.6 pp absolute gain; detector F1 is strongly positively correlated with downstream correction gain.

Limitations¶

Experiments restricted to counts in $\{1,\ldots,9\}$; scalability to larger counts is untested.
Synthetic datasets use white backgrounds and non-overlapping objects; real-world clutter, occlusion, and texture are only partially addressed through CountBench.
Causal steering requires sweeping $\alpha$ and is sensitive to scale; excessive steering is destructive, limiting direct use as a practical intervention.
Self-correction depends on a correction re-prompt that may itself introduce errors or fail on harder samples.
Detector performance varies substantially across models (near-perfect on InternVL2 vs. weaker on Qwen3-VL), suggesting correction gains are uneven.
Analysis covers four models from two families; generalization to other VLM architectures is assumed but not verified.

Relevance to Vision-Language Models¶

This work provides one of the first mechanistic accounts of why VLMs fail at counting, moving beyond behavioral benchmarks to locate the failure in a misalignment between internal representations and the generation pathway — a finding with broad implications for understanding VLM reliability and grounding. The multi-probe + SVCCA methodology is immediately reusable for studying other quantitative reasoning failures (e.g., spatial relations, attribute binding) in VLMs. The inference-time detector-guided self-correction is a practically useful technique requiring no fine-tuning, making it applicable to black-box-adjacent settings. For researchers studying hallucination and faithfulness in VLMs, the result that the correct answer is often latent but misread during decoding reframes the problem from perception to alignment between representation and language head.