Skip to content

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

🕒 Published (v1): 2026-07-10 14:09 UTC · Source: Arxiv · link

Why this paper was selected

Test-time scaling on small open VLMs—transfers TTS gains beyond frontier models; highly actionable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper evaluates test-time scaling (TTS) strategies for small open VLMs (≤7B parameters) on EXAMS-V, a multilingual visual multiple-choice benchmark, finding that per-chain token budget and policy quality dominate over structured search methods. PRM-guided beam search underperforms flat self-consistency at 8.7× the cost, and neither a generative critic nor a trained PRM beats majority vote. The best configuration (Qwen3.5-4B, SC-\(N\)=16, 2k-token budget, guided repair) reaches 84.1% on the held-out test split, ranking first on the ImageCLEF 2026 Visual MCQ leaderboard.

Problem

Whether inference-time scaling techniques proven for large text-only LLMs transfer to small open VLMs (≤7B) under a single-GPU budget remains unclear, particularly for multilingual multimodal benchmarks where discriminative PRMs trained on mathematics may not generalize and chain truncation masquerades as reasoning failure.

Method

Two main pipelines are compared across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B on the full EXAMS-V validation set (4,651 questions, 11 languages):

Describe-then-reason (DTR) with PRM-guided beam search. \(N\) image-grounded descriptions are sampled at \(T\)=0.7; each description seeds either flat chain sampling (\(M\) chains, no image) or PRM-BAS beam annealing (initial beam \(B_0\)=4, \(B\)=2 continuations per surviving beam, threshold \(\tau\)=0.05, depth \(d\)=6) scored by Qwen-VL-PRM-7B. Post-hoc selectors (majority vote, training-free generative critic scored on three rubric axes, discriminative PRM in one-shot mode) pick the final answer from the candidate pool. A skip rule passes high-confidence majorities (≥5/8) through unchanged.

Flat self-consistency (SC). \(N \in \{8, 16\}\) chains are sampled at \(T\)=0.7 with max_new_tokens ∈ {1024, 2048} from an MMMU-standard CoT prompt; majority vote with log-probability tie-breaking selects the answer.

Guided parse repair. When all chains for a question fail regex extraction, each chain's reasoning is concatenated with \n\nAnswer: and a single token is decoded under guided_choice over {A,B,C,D,E}, forcing commitment at zero additional reasoning cost.

Inference uses vLLM with batched parallel decoding and automatic prefix caching on a single A100-40 GB (matching the shared task's A40 constraint). Statistical comparisons use McNemar's paired test with Bonferroni correction.

Key Contributions

  • Empirical demonstration that per-chain token budget (\(+3.7\) pp from 1k→2k tokens) far dominates chain count (\(+0.15\) pp from \(N\)=8→16) in small-VLM TTS.
  • Evidence that PRM-guided beam search underperforms flat self-consistency by \(-0.39\) pp at 8.7× the compute on multilingual visual MCQ, with P(+) saturation (0.962 on descriptions, 0.849 on reasoning) collapsing search diversity.
  • Replication that neither a training-free generative critic nor a cross-model discriminative PRM beats majority vote across two policies and two pool strengths.
  • Identification of parseability as a confounding artifact: an early prompt format caused ∼9% chain parse failures, inflating apparent reasoning failures; MMMU-standard closer and guided repair eliminate this artifact.
  • Stratified analysis showing TTS gains concentrate on mid-accuracy languages where chain errors are independent, while high-correlation-error languages (Chinese) resist SC because the vote amplifies shared mistakes.
  • First-place result on the ImageCLEF 2026 Visual MCQ leaderboard at 84.06% on the held-out test split.

Results

  • Policy swap dominates all TTS methods: Qwen2.5-VL-7B → Qwen3.5-4B at matched SC-\(N\)=8 (1k): \(+11.4\) pp (66.42% → 77.81%).
  • Token budget: SC-\(N\)=8 at 2k vs. 1k on Qwen3.5-4B: \(+3.7\) pp (81.49% vs. 77.81%).
  • Chain count: SC-\(N\)=16 vs. \(N\)=8 (both 2k): \(+0.15\) pp (81.64% vs. 81.49%); within sampling noise.
  • PRM-BAS vs. flat SC: 80.74% vs. 81.13% on same 4,319 questions (\(-0.39\) pp), at 8.7× compute cost.
  • Selectors (Q3.5 pool): Generative critic \(-0.47\) pp; PRM (1-shot) \(-0.09\) pp; both non-significant improvements over majority vote.
  • Full TTS progression on Qwen3.5-4B: zero-shot 57.11% → CoT 69.66% → SC-\(N\)=8 (1k) 77.81% → SC-\(N\)=8 (2k) 81.49% → SC-\(N\)=16 (2k) 81.64%.
  • Test split (held-out): 84.06% overall; English 85.6%, Chinese 78.1%; first on leaderboard across all six test languages.
  • Guided repair: eliminates 100% of parse-fail questions (82 on test split) at one decoded token per chain.

Limitations

  • Both policies are from the same family (Qwen), so behavior specific to Qwen versus small open VLMs generally cannot be separated; the parseability effect depends on how a specific model emits answer letters.
  • Ablations on expensive configurations used a 200-question dev subset with ±7 pp Wilson CIs, precluding ranking claims; only headline findings are confirmed on val-full (\(n\)=4,651).
  • The PRM null may be an artifact of poor per-step calibration on multilingual non-mathematical content rather than a fundamental ceiling; better-calibrated PRMs remain untested.
  • No fine-tuning, few-shot exemplars, or test-time image augmentation were explored, all of which may yield further gains.
  • The P(+) asymmetry between perception (0.962) and reasoning (0.849) steps is diagnosed but not addressed; decoupling perception from reasoning is left as future work.
  • EXAMS-V lacks content-type annotations, so subject is used as a proxy for visual complexity.

Relevance to Vision-Language Models

This paper provides a controlled empirical study of how established LLM TTS techniques transfer to small open VLMs under realistic resource constraints, directly informing practitioners deploying sub-7B multimodal models. The key finding that token budget is a stronger lever than search width or selection sophistication challenges the prevailing assumption that structured search methods generalize from math benchmarks to multilingual, mixed-domain visual MCQ. The PRM generalization null — caused by P(+) saturation and loss of discriminative signal outside mathematics — is a concrete diagnostic for why multimodal PRMs fail out-of-distribution, pointing toward a gap in current VLM reward modeling. The parseability artifact analysis is methodologically significant for anyone evaluating VLM reasoning via chain-of-thought, as extraction failures can silently masquerade as reasoning failures and inflate apparent TTS gains.