ProxyThinker: Test-Time Guidance through Small Visual Reasoners¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; small proxy VLMs guide test-time reasoning of large VLMs cheaply

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ProxyThinker is a training-free inference-time method that transfers slow-thinking reasoning behaviors from small, RFT-trained visual reasoners to large base VLMs by adding a logit delta — the difference between a small RFT expert and its base counterpart — to the large model's output distribution at each decoding step. This elicits self-verification and self-correction behaviors in models that never underwent reinforcement fine-tuning. The approach is implemented on vLLM with asynchronous tensor-parallel scheduling, achieving a 38× speedup over prior decoding-time steering methods.

Problem¶

Reinforcement fine-tuning (RFT) with verifiable rewards is the dominant way to induce "slow-thinking" reasoning in VLMs, but it is prohibitively expensive for models beyond 7B parameters (requires multiple model copies for PPO/GRPO, alternating rollout/optimization phases). This leaves large 32B–72B base VLMs without accessible slow-thinking capabilities. Prior decoding-time steering methods exist but are not designed for multimodal reasoning transfer and are slow due to naive pipeline parallelism.

Method¶

ProxyThinker modifies the next-token distribution of a large base model \(\Psi\) at each decoding step \(t\) using a logit delta derived from two small auxiliary models — an RFT-trained reasoning Expert \(\psi_1\) and a non-RFT Amateur \(\psi_0\) (both ~7B):

\[p(x_t \mid x_{<t}, \mathcal{I}) = \text{softmax}\!\left[z_\Psi(x_t \mid x_{<t}, \mathcal{I}) + \alpha \cdot \bigl(z_{\psi_1}(x_t \mid x_{<t}, \mathcal{I}) - z_{\psi_0}(x_t \mid x_{<t}, \mathcal{I})\bigr)\right]\]

The delta \(z_{\psi_1} - z_{\psi_0}\) isolates the probability mass shift induced by RFT (e.g., elevated probability for tokens like "Wait", "However", "But") while the Amateur model calibrates away noise from the expert's raw logits. All three models autoregressively share the same growing context prefix. The Amateur is critical: without it, the method reduces to contrastive decoding, which underperforms. Implementation uses vLLM with tensor parallelism applied to the large base model and asynchronous separate tensor-parallel process groups for the small models, with logit synchronization via collective communication before sampling.

Key Contributions¶

Training-free test-time steering that transfers RFT-induced reasoning behaviors (backtracking, verification, sub-goal branching) from a 7B visual reasoner to 32B/72B base VLMs.
Formal derivation of logit-delta decoding for multimodal settings, distinguishing the Amateur's calibration role from contrastive decoding.
vLLM-based asynchronous multi-model inference implementation achieving 38× wall-clock speedup over prior decoding-time methods.
Quantitative analysis of emerging reasoning patterns (backtracking +137.4%, verification +31.4%, sub-goal +0.6% vs. base on MathVision 3040 samples via GPT-4o-mini judging).
Pass@k analysis showing ProxyThinker sits between base and expert curves, preserving diversity better than full RFT while improving greedy-decode accuracy.

Results¶

MathVision: ProxyThinker-32B (OpenVLThinker-7B expert) raises Qwen2.5-VL-32B-Instruct from 38.4% → 40.8%, surpassing full RFT model VL-Rethinker-32B (40.5%); best \(\alpha\) yields 40.3% (vs. RFT ceiling 40.5%).
MathVerse: ProxyThinker-32B (VL-Rethinker-7B) reaches 55.3% (+1.5pp over 53.8% base); best \(\alpha\) gives 57.2%, matching VL-Rethinker-32B ceiling (56.9%).
MMMU-Pro: Consistent +2–3pp gains across 32B expert combinations.
R1-OnevisionBench: +0.3 to +3.6pp depending on expert choice.
72B scale: Consistent but smaller gains; VL-Rethinker-7B expert achieves best Δ=+2.7% average relative improvement.
Inference speed: 38× faster than prior open-source decoding-time steering baselines.
Baselines: Qwen2.5-VL-32B/72B-Instruct (base), VL-Rethinker-32B/72B (full RFT ceiling), OpenVLThinker-7B, ThinkLite-VL-7B, VL-Rethinker-7B (small experts).

Limitations¶

Requires running three concurrent models at inference, increasing GPU memory footprint compared to single-model decoding even with the optimized scheduler.
Performance gains diminish at 72B scale, suggesting the logit delta signal from a 7B expert has bounded influence on much larger models.
Underperforming or mismatched RFT experts can introduce negative transfer on some benchmarks (e.g., MathVision -1.9pp with OpenVLThinker-7B at 72B scale).
Expert and base models must share the same tokenizer/vocabulary for logit-space alignment; cross-family transfer is not explored.
Pass@k analysis shows reduced reasoning diversity relative to the bare base model for \(k \geq 2\), meaning ProxyThinker partially inherits the diversity-reduction side-effect of RFT.
Only evaluated on mathematical and multi-disciplinary reasoning; generalization to open-ended visual tasks (VQA, captioning) is not demonstrated.

Relevance to Vision-Language Models¶

ProxyThinker addresses a central scalability bottleneck in LVLM reasoning research: the gap between the cost of RFT and the size of models practitioners actually deploy (32B–72B). By showing that a 7B RFT expert's logit delta can steer a 72B base model toward self-verification behaviors without any weight updates, the paper strengthens the hypothesis that RFT amplifies latent base-model capabilities rather than encoding new knowledge — a claim with significant implications for how the VLM training community should think about reasoning emergence. The approach is directly complementary to ongoing work on test-time compute scaling and model merging in VLMs, and the vLLM implementation makes it practically deployable. For researchers tracking VLMs, this is a lightweight, effective alternative to full RFT that can be applied to any existing large base model with a small RFT counterpart.