Skip to content

Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TON (Think-or-Not) is a two-stage training framework that teaches VLMs to selectively skip chain-of-thought reasoning for easy queries while retaining it for hard ones. It combines a "thought dropout" SFT cold-start with GRPO-based RL exploration, reducing completion length by up to 90% without accuracy loss—and sometimes improving it by up to 17 points.

Problem

Standard GRPO-based RL post-training forces VLMs to always generate full reasoning traces regardless of question difficulty, producing redundant token generation and inflated compute cost. Prompting-based workarounds (hybrid-thought prompts) fail because models default to full thinking due to pre-training inertia; selective reasoning cannot emerge from prompt engineering alone—it must be explicitly trained as a format-following skill.

Method

TON is a two-stage pipeline built on Qwen-2.5-VL (3B/7B):

Stage 1 — Thought Dropout SFT. High-quality reasoning traces are generated via reverse thinking: the base model is prompted with (image, question, ground-truth answer) to produce the intermediate thought, avoiding reliance on external APIs. During SFT, a thought_dropout function randomly replaces the <think> content with an empty "\n\n" delimiter at a configurable probability (default 50%). This instills the <think>\n\n</think><answer>…</answer> format as a valid output mode, serving as a cold start for selective reasoning.

Stage 2 — GRPO with selective exploration. Standard GRPO sampling is applied with no additional length-penalty reward. Because Stage 1 unlocks the empty-think format, the model now samples from both "think" and "non-think" trajectories, expanding response distribution diversity (the α/β importance-weight terms in the GRPO objective). Reward is purely outcome-based: binary discrete matching for classification/counting/math tasks, and point-in-box / distance-threshold continuous matching for GUI navigation tasks. The model freely learns when to skip reasoning by optimizing task-aware rewards.

Key Contributions

  • First systematic study of "when to think" (selective reasoning) in VLMs, distinct from prior work on "how to think" or reasoning compression.
  • Thought dropout: a trivially implementable SFT augmentation (a single if branch) that activates skip-think as a learnable behavior.
  • Reverse thinking: self-supervised cold-start thought generation from (V, Q, S*) triples without external model APIs.
  • Empirical demonstration that the skip ratio increases monotonically with reward during training, indicating the model internalizes task difficulty adaptively.
  • Evidence that random dropout outperforms difficulty-aware (heuristic) dropout, showing hand-crafted difficulty signals introduce noise.

Results

  • CLEVR (3B): GRPO → 93.5% acc / 227 tokens; TON → 98.5% acc / 28 tokens. +5 acc, −87% length, −47 min training time.
  • GeoQA (3B): GRPO → 37% acc / 272 tokens; TON → 51% acc / 96 tokens. +14 acc, −65% length, −46 min.
  • GeoQA (7B): GRPO → 44% acc / 1069 tokens; TON → 61% acc / 112 tokens. +17 acc, −90% length, −29 min.
  • AITZ mobile navigation (3B): task-level output length reduced from 3,664 to 922 tokens (−75%); OOD avg exact accuracy maintained (0.59 vs 0.59).
  • AITZ (7B): length 3,272 → 908 tokens (−72%); OOD avg exact accuracy 0.45 vs 0.45 for GRPO.
  • GSM8K (LLM text-only): TON reduces response length significantly while maintaining high accuracy (exact numbers not reported in text excerpt, shown in Figure 6b).
  • Super-CLEVR (OOD): TON generalizes to unseen 3D object counting domain.

Limitations

  • Skip ratio behavior and optimal dropout probability (20%/50%/80%) are task-dependent; no principled criterion for selecting the dropout rate is provided—ablation suggests lower is better but the mechanism is underspecified.
  • Reverse thinking quality is bounded by the base model's capability; for tasks where the base model cannot reliably reason backward from the answer, cold-start thought quality may degrade.
  • Reward design remains manually task-specific (discrete vs. continuous matching must be engineered per benchmark); no unified automatic reward is proposed.
  • All experiments use Qwen-2.5-VL-3B/7B only; generalization to other VLM families is unverified.
  • The 90% token reduction is measured at training endpoint, not under real deployment serving conditions (e.g., batched inference, KV-cache dynamics not analyzed).

Relevance to Vision-Language Models

TON directly addresses the efficiency bottleneck introduced by RL-based reasoning enhancement (GRPO/R1-style) in VLMs—a now-standard post-training paradigm. It shows that selective reasoning is a distinct, trainable capability orthogonal to reasoning quality, with implications for any VLM deployment where inference cost matters. The thought-dropout technique is lightweight and composable with existing GRPO pipelines, making it practically relevant for practitioners fine-tuning VLMs on domain-specific tasks. The AITZ results are particularly significant for the GUI/agentic VLM subfield, where multi-step reasoning overhead compounds across action steps.