On Test-Time Scaling for Vision-Language Models¶

🕒 Published (v1): 2026-06-27 11:12 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Test-time scaling is the hottest topic; timely application to VLMs is high-impact

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents the first comprehensive study of conventional LLM test-time scaling (TTS) methods applied to vision-language models across 13 models, 9 methods, and 6 benchmarks. Contrary to prior LLM findings, small LVLMs (2B–8B) benefit most from TTS (up to ~30% improvement), often reaching or surpassing larger-model baselines. The authors also show TTS degrades perceptual tasks and that visual information is encoded only in an early window of the reasoning chain.

Problem¶

Prior work on LLMs reports that TTS is ineffective for small models (<10B), and several VLM-specific studies claim that even Chain-of-Thought prompting fails for LVLMs, leading researchers to develop bespoke techniques (e.g., Compositional CoT). Whether, when, and to what extent standard LLM TTS methods transfer to LVLMs across model scales and benchmarks had not been systematically studied.

Method¶

The authors evaluate 9 zero-shot (no additional training) test-time scaling methods on 13 instruction-tuned LVLMs from the Qwen-2.5-VL, Qwen-3-VL, InternVL-3.5, and Molmo2 series (2B–72B). The methods are: CoT, Structured CoT, Plan-and-Solve, Self-Consistency (b=5 runs), Self-Aggregation (b=5 runs, then concatenates all chains), Self-Refinement (k=3 rounds), Describe-Answer (caption→answer), Compositional CoT (scene graph→answer), and Prompt Repetition (repeat question). Each is implemented by appending a think-prompt before the post-prompt. Reliable answer extraction uses a fallback second pass if parsing fails. Maximum token length is 1024. Benchmarks span reasoning (WeMath, LogicVista), perception (RealWorldQA, A-OKVQA), hallucination (HallusionBench), and mixed (MMStar). A reduced 300-token budget ablation tests token-length effects, and attention analysis tracks image-token contribution over the generation.

Key Contributions¶

First large-scale study of standard LLM test-time scaling methods applied to LVLMs, spanning 13 models × 9 methods × 6 benchmarks
Finds that small, capable instruction-tuned LVLMs benefit most from TTS (up to ~30% gain), often reaching or exceeding the baseline performance of models 8× larger — directly contradicting the LLM literature
Demonstrates that TTS degrades performance on perception-only tasks because VLMs "overthink": they hallucinate, make wrong assumptions, and accumulate chain errors when given more tokens than needed
Shows via attention analysis that visual information is encoded in a short early window of the reasoning chain, after which attention to image tokens decays and the chain is dominated by text-only reasoning

Results¶

Qwen3-VL-2B: Self-Consistency boosts WeMath from 35.46% → 64.25% (≈30% absolute gain), matching larger-model baselines
Qwen3-VL-4B with simple CoT surpasses Qwen3-VL-32B baseline on HallusionBench, WeMath, and LogicVista (e.g., +13% on LogicVista)
Prompt Repetition yields ~+4% on WeMath for Qwen3-VL-2B by creating a bidirectional-attention effect
CoT offers the best accuracy–runtime tradeoff across all methods (single forward pass, modest overhead)
On perception benchmarks (RealWorldQA, A-OKVQA), TTS degrades or fails to improve; a 300-token budget ablation confirms shorter generations are better for perceptual tasks
Results generalize across Qwen-3-VL, InternVL-3.5, and Molmo2 series; the failure of prior VLM TTS work is attributed to poor instruction-following in older models (LLaVA, SmolVLM2)
On closed-source GPT-5.2, perceptual degradation is less pronounced but still present

Limitations¶

Study is limited to open-source LVLMs (Qwen, InternVL, Molmo2); closed-source model coverage is minimal (only GPT-5.2 on two benchmarks)
Maximum token length capped at 1024 — effects at much longer reasoning chains are not explored
Self-Consistency uses only b=5 samples; larger ensembles might yield different findings
The "overthinking" explanation for perceptual degradation is supported by attention analysis but remains correlational
All methods are zero-shot prompting — trained TTS approaches (e.g., reinforcement-learning-based "thinking models") are explicitly excluded

Relevance to Vision-Language Models¶

This paper directly challenges the emerging consensus that VLM-specific TTS methods are necessary and that standard LLM TTS fails for VLMs. For practitioners tracking this topic, it provides a practical accuracy–compute Pareto analysis showing that small VLMs (2B–8B) with cheap inference-time scaling can rival much larger models, which has direct implications for deployment cost and latency. The finding that VLMs "overthink" on perception tasks also offers a clear guideline: gate TTS on task type.