Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models

🕒 Published (v1): 2026-05-29 01:06 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

Mrinmaya Sachan (ETH); ICML 2026; systematic study of test-time compute in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Test-time compute (TTC) strategies from the LLM literature transfer poorly to VLMs due to low output diversity: majority voting yields only 2–4% gains under CoT and zero gain under direct answering, while feature-based heuristics fail entirely. The paper proves that voting improvement is monotonically decreasing in inter-prediction dependency, and proposes Entropy-based TTC (ETTC), which selects the minimum-entropy prediction in multi-model ensembles rather than counting votes, consistently outperforming both voting and the best single model.

Problem

Existing TTC methods (majority voting, feature-based Best-of-N) were designed for LLMs and rely on assumptions that break in the visual domain: (1) linguistic heuristics (pivot words, CoT length) that correlate with correctness in text tasks do not reflect visual understanding quality; (2) single-model decoding produces low-diversity outputs in VLMs due to a perception bottleneck—visual grounding must succeed before any meaningful reasoning variation can emerge; (3) standard majority voting in multi-model ensembles treats all models equally, allowing correlated weaker models to outvote a confident stronger one.

Method

Single-model analysis. Seven VLMs are queried \(N=16\) times per question (stochastic decoding) on six multiple-choice visual QA benchmarks. Prediction diversity is quantified by (a) average pairwise normalized mutual information (NMI) over raw answer options and (b) average pairwise correctness correlation \(\rho\): $\(\rho(Z, Z') = \frac{E[ZZ'] - p^2}{p(1-p)}, \quad \overline{\text{NMI}} = \frac{2}{U(U-1)}\sum_{u<v}\text{NMI}(X_u; X_v)\)$ Theorem 1 proves \(\Delta A_\text{MV}(U)\) is monotonically decreasing in both \(\rho\) and NMI; empirical scatter plots across 7 models × 6 datasets confirm the negative correlation.

ETTC (Entropy-based TTC). In the multi-model ensemble setting, each model \(u\) produces a predictive distribution \(p_u(\cdot) \in \Delta^{K-1}\). ETTC selects the prediction with minimum normalized entropy: $\(\hat{H}_u = -\frac{1}{\log K}\sum_{k=1}^K p_u(k)\log p_u(k), \quad \hat{Y}_{\min H} = \hat{y}_{u^*}, \quad u^* = \arg\min_u \hat{H}_u\)$ In the single-model case, averaging distributions over rounds makes ETTC equivalent to majority voting. Theorem 2 shows that under an entropy-accuracy monotonicity assumption (lower entropy \(\Rightarrow\) higher \(P(\hat{y}=Y)\)), ETTC satisfies \(A_{\min H} \geq A_\text{MV}(\lambda)\) for all correlation levels \(\lambda \in [0,1]\), with strict inequality when \(\lambda > 0\) and the correlated-bloc accuracy \(\bar{c} < c^*\) (best-model accuracy).

Ensemble configurations. (1) Similar-size cross-family: Qwen2.5-VL-7B, LLaMA-3.2-11B, Gemma-3-12B, Pixtral-12B. (2) Same-family varied-size: Qwen2.5-VL 3B/7B/32B/72B.

Key Contributions

• Systematic empirical study of TTC across 7 VLMs and 6 benchmarks under both direct-answer and CoT prompting conditions.
• Formal proof (Theorem 1) that majority voting improvement is monotonically decreasing in pairwise prediction dependency (NMI and \(\rho\)), with empirical validation.
• Identification that \(U=12\)–16 decoding samples suffice for stable dependency estimation.
• ETTC: a training-free, model-agnostic aggregation rule that reduces to majority voting in the single-model case and provably dominates voting in multi-model ensembles (Theorem 2).
• Demonstration that smaller models can synergistically enhance larger ones via ETTC, achieving gains unavailable to standard voting.

Results

• Feature-based methods: zero consistent improvement in all settings; often indistinguishable from vanilla baseline.
• Majority voting, direct answer: negligible improvement (<1%) across all 6 benchmarks averaged over 7 VLMs.
• Majority voting, CoT: consistent +2–4% average gains over greedy, but modest due to high inter-sample correlation.
• ETTC vs. voting, cross-family ensemble (Qwen-7B + LLaMA + Gemma + Pixtral):
• Average accuracy: ETTC 66.56% vs. voting 63.75% vs. best single model (Qwen-7B) 59.91%.
• MathVista: ETTC 75.93% vs. voting 68.33% vs. best single 65.03%.
• MathVision: ETTC 35.57% vs. voting 32.05% vs. best single 31.84%.
• ScienceQA: ETTC 85.28% vs. voting 85.52% (near-parity) vs. best single 78.32%.
• ETTC surpasses the best individual model on average across all six benchmarks.
• Dependency convergence: NMI and \(\rho\) stabilize at \(U=12\) samples (Fig. 2, Qwen-7B).
• Negative correlation between \(\Delta A_\text{MV}(16)\) and NMI/\(\rho\) confirmed across all 7 models × 6 datasets (Fig. 3); smaller models (Qwen-3B, LLaMA) show higher diversity and larger voting gains than larger models (Qwen-72B, Pixtral).

Limitations

• Restricted to multiple-choice QA; entropy-based selection is straightforward for discrete answer options but may not generalize to open-ended generation without post-hoc extraction.
• Entropy-accuracy monotonicity (Assumption 1) holds in aggregate but may fail instance-by-instance; ETTC inherits this fragility at the example level.
• Experiments cover only open-source VLMs; closed-source models (GPT-4V, Gemini) are excluded, so token-probability access required by ETTC limits applicability to white-box or logit-accessible models.
• The cross-family ensemble requires running \(M\) full models; inference cost scales linearly with ensemble size.
• Dependency metrics (NMI, \(\rho\)) require labeled data or a held-out subset for calibration in practice.

Relevance to Vision-Language Models

This paper provides the first rigorous TTC analysis specifically for VLMs, establishing that the perception bottleneck suppresses output diversity in a way that fundamentally limits voting-based inference scaling—a finding with direct implications for how the community should allocate test-time compute budgets. ETTC directly addresses the heterogeneous-capability problem arising in VLM ensembles, where architecture and scale differences create exploitable confidence signals, enabling smaller open-source VLMs to complement larger ones in a principled way. The theoretical framework (diversity ↔ voting gain) offers a pre-hoc diagnostic—compute NMI/ρ on a small sample to predict whether TTC will help—that is broadly applicable to any new VLM or benchmark. This positions ETTC as a lightweight plug-in over any collection of VLMs, relevant to researchers designing multi-model pipelines, benchmark evaluation protocols, or resource-constrained deployment stacks.