Skip to content

SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ThinkLite-VL is a family of reasoning VLMs trained via reinforcement fine-tuning (RFT) on MCTS-filtered subsets of just 11k (7B) and 7.5k (72B) samples, achieving state-of-the-art results on MathVista (75.1 for 7B, 79.7 for 72B) without any knowledge distillation. The core insight is that sample difficulty—operationalized as the number of MCTS iterations a base VLM needs to solve a problem—is the decisive signal for RFT effectiveness, not data volume.

Problem

RFT for VLMs is less effective than for pure LLMs due to the modality gap, and current high-performing reasoning VLMs depend on expensive knowledge distillation from larger/closed-source models and large SFT datasets (100k–260k samples). Scaling training data indiscriminately does not substitute for appropriately challenging examples; no reliable, scalable method existed to quantify sample difficulty for VLMs.

Method

Starting from a 70k open-source pool (Geometry3K, GeoQA, Geos, FigureQA, ScienceQA, OK-VQA, IconQA, TabMWP; math, image understanding, chart comprehension), MCTS is repurposed as an offline difficulty estimator:

  1. MCTS simulation: For each training instance, the base VLM (Qwen2.5-VL-7B-Instruct or 72B) acts as the policy πθ. MCTS runs selection (visit-count UCB), expansion (k=3 candidate steps via temperature=0.5 decoding), and simulation (rollout to final answer) iteratively up to 50 iterations. A Qwen2.5-7B-Instruct critic judges answer correctness.
  2. Difficulty metric: K = number of MCTS iterations to reach the first correct answer. Unsolved samples (K > 50) are retained; samples with K ≤ 5 are discarded as too easy.
  3. Filtered subsets: 11k samples (7B) and 7.5k samples (72B) are selected (K > 5 or unsolved).
  4. RFT via GRPO: GRPO is applied directly on the filtered set with no SFT or distillation stage; 32 rollouts per prompt; outcome-based reward.
  5. Multiple-choice questions are reformatted to open-ended to prevent answer-guessing and improve difficulty signal reliability.

Key Contributions

  • MCTS repurposed as an offline sample-difficulty estimator for VLM RFT, using iteration count rather than a learned reward model.
  • Demonstration that appropriately challenging data (not volume) is the key driver of RFT gains in VLMs.
  • ThinkLite-VL-7B and ThinkLite-VL-72B achieve SoTA across eight benchmarks using 11k and 7.5k samples respectively—an order of magnitude less data than competitors—without SFT or knowledge distillation.
  • Empirical finding that model-specific difficulty curation does not transfer across scales (7B-selected data trains 72B worse, and vice versa), motivating compute scaling per model.
  • Full open-source release of models and MCTS-filtered training sets.

Results

  • ThinkLite-VL-7B: 75.1 on MathVista (new SoTA among all 7B models; beats GPT-4o, o1, Qwen2.5-VL-72B); average 64.18 across 8 benchmarks vs. base 59.69 (+4.49 absolute, +7% relative).
  • ThinkLite-VL-72B: 79.7 on MathVista (SoTA among all open-source VLMs); average 72.67 vs. base 68.25 (+4.42 over open-source SoTA).
  • vs. random selection (11k): MCTS filtering yields +3.29 average over ThinkLite-VL-Random11k (64.18 vs. 60.89) and +3.06 over Random7.5k (72.67 vs. 69.91).
  • vs. full 70k training: ThinkLite-VL-7B (11k) outperforms training on the entire 70k set (64.18 vs. 63.13 average), demonstrating quality over quantity.
  • vs. self-consistency filtering (offline, 23k): MCTS 11k (64.18) beats self-consistency 23k (63.15) despite using half the data.
  • vs. online self-consistency: MCTS offline selection (64.18 at 7B; 72.67 at 72B) outperforms online self-consistency filtering (62.34 at 7B; 70.12 at 72B).
  • Cross-scale transfer ablation: Training 7B on 72B-selected data drops average to 61.24 vs. 64.18; training 72B on 7B-selected data drops to 70.24 vs. 72.67.

Limitations

  • MCTS-based difficulty estimation requires running up to 50 MCTS iterations per sample over the full 70k pool, which is computationally expensive at inference time before training begins (compute scaling as a prerequisite).
  • Optimal difficulty threshold (K > 5) is not theoretically derived; ablations show sensitivity to this cutoff.
  • Difficulty sets are model-specific and do not transfer across scales, requiring separate MCTS runs for each model family/size.
  • Evaluation is limited to visual mathematical and scientific reasoning benchmarks; generalization to other VLM capabilities (generation, grounding, long-form QA) is untested.
  • No analysis of how MCTS-filtered data distribution changes with different base models or domains outside the three covered.

Relevance to Vision-Language Models

This paper directly addresses a core bottleneck in reasoning VLM training—the dependence on knowledge distillation and large SFT datasets—by showing that difficulty-aware data curation alone can match or surpass distillation-heavy pipelines. For the VLM community, it establishes a principled, model-intrinsic difficulty signal via MCTS, complementing lines of work on process reward models and chain-of-thought self-improvement. The finding that model-specific difficulty calibration is essential challenges the common practice of sharing reasoning datasets across model families. The data-efficiency result (11k vs. 200k+ for comparable SoTA) has direct implications for resource-constrained research and democratizing VLM reasoning training.