Skip to content

Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TON (Think-or-Not) is a two-stage training framework that teaches vision-language models to selectively skip chain-of-thought reasoning when it is unnecessary. Using "thought dropout" during SFT followed by GRPO-based RL, TON reduces completion length by up to 90% without degrading—and sometimes improving—task accuracy. The method is evaluated across counting, math, GUI agent navigation, and text-only benchmarks.

Problem

Standard GRPO-based RL post-training forces VLMs to always generate full reasoning traces regardless of question difficulty, causing redundant token usage and inflated training/inference cost. Prompt-based workarounds fail because models default to generating complete thoughts even when instructed otherwise, indicating that selective reasoning must be trained as an explicit skill.

Method

TON is a two-stage pipeline applied on top of Qwen-2.5-VL (3B/7B):

  1. SFT with Thought Dropout: High-quality reasoning traces are first generated via "reverse thinking" (conditioning the base model on the question, image, and ground-truth answer to produce a thought). During SFT, each trace is randomly replaced with an empty \n\n delimiter at a configurable dropout probability (default 50%), teaching the model that the <think>…</think> block is optional format.

  2. GRPO Stage: Standard GRPO is applied with only task-outcome rewards (binary discrete matching for classification/counting/math; coordinate-hit continuous matching for grounding/navigation). The SFT cold start ensures the model's output distribution includes non-think samples, expanding the diversity of GRPO rollouts beyond the vanilla think-only distribution and allowing RL to discover when skipping pays off.

Key Contributions

  • First work to frame "when to think" (versus "how to think") as a trainable skill for VLMs.
  • Thought dropout: a single-line SFT augmentation that seeds selective reasoning without requiring external reward signals or difficulty labels.
  • Demonstration that random dropout outperforms difficulty-aware (heuristic) dropout, suggesting hand-crafted difficulty labels introduce noise.
  • Empirical evidence that skipping reasoning can improve accuracy (up to +17% on GeoQA), challenging the assumption that longer reasoning is always better.
  • Generalization to text-only LLMs (GSM8K) and agentic multi-step mobile navigation (AITZ).

Results

  • CLEVR (3B): TON 98.5% vs. GRPO 93.5% (+5.0%); length 28 vs. 227 (−87%); training time −47 min.
  • GeoQA (3B): TON 51% vs. GRPO 37% (+14%); length 96 vs. 272 (−65%); training time −46 min.
  • GeoQA (7B): TON 61% vs. GRPO 44% (+17%); length 112 vs. 1069 (−90%); training time −29 min.
  • AITZ OOD (3B): Average exact accuracy 0.59 (TON) vs. 0.59 (GRPO); task-level output length 922 vs. 3664 tokens (−75%).
  • AITZ OOD (7B): Average exact accuracy 0.45 (TON) vs. 0.23 (GRPO, +0.22); task-level length 908 vs. 3272 (−72%).
  • GSM8K: TON significantly reduces response length while maintaining accuracy comparable to GRPO (exact numbers not given in text).
  • Skip ratio increases monotonically with training reward, and non-think outputs maintain comparable per-thought length to vanilla GRPO when reasoning is engaged.

Limitations

  • Experiments restricted to Qwen-2.5-VL-3B/7B; generality to other VLM families not demonstrated.
  • Dropout probability is a static hyperparameter set before training; the paper proposes dynamic scheduling conceptually but does not implement it.
  • AITZ OOD results on 7B show mixed gains (e.g., Google Apps exact accuracy drops −0.09), indicating selective reasoning does not uniformly help in all navigation sub-domains.
  • Thought quality depends on the base model's reverse-thinking capability; very weak base models may produce low-quality cold-start traces.
  • No wall-clock inference latency benchmarks reported—only training time and token count reductions.

Relevance to Agentic AI / LLM Agents

For long-horizon GUI and mobile agents like those on AITZ, token efficiency is a first-class concern: every intermediate action step generates reasoning, so multiplicative token savings translate directly to faster and cheaper deployments. TON's 70–75% task-level token reduction on AITZ without accuracy loss is directly actionable for scaling agent systems. The selective reasoning paradigm also connects to broader questions of metacognitive control in agents—deciding when to invoke deliberate planning versus reactive response—which is increasingly recognized as critical for efficient tool-calling and agentic workflows. The two-stage SFT→GRPO pattern is immediately composable with existing RL-based agent training pipelines.