Skip to content

EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction Evolution

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvolvedGRPO addresses reward convergence in GRPO-based LVLM training by progressively generating harder multi-modal questions via adversarially-trained instruction evolution. A question-generator model and answer model are jointly trained via GRPO, with the generator rewarded for producing problems the answer model cannot solve, and difficulty increased via curriculum scaling of instruction-chain length. The method achieves state-of-the-art among open-source 7B reasoning models on multi-modal math benchmarks.

Problem

Applying GRPO to LVLMs causes intra-group reward variance to collapse as training progresses: the model memorizes visual+textual patterns from fixed training data, causing all group responses to converge to similar rewards. This halts gradient updates and degrades chain-of-thought reasoning. Simple samples get uniformly high rewards (no signal), while hard samples get uniformly low rewards (no learning of incremental improvements). Existing fixes either distill from stronger external models (capping the performance ceiling) or require costly human data collection.

Method

EvolvedGRPO maintains two model instances—a question generator Q and an answer model A—both initialized from the same LVLM checkpoint (Qwen2.5-VL-7B-Instruct) and trained alternately via GRPO.

Instruction evolution: Starting from a seed sample (q₀, a₀, v₀), Q generates a chain of k editing instructions applied sequentially to produce (qₖ, aₖ, vₖ). Two instruction types are used: - Image editing instructions (Iᵥ): Q produces prompts for an external diffusion model (Flux.Kontext) to modify the image while preserving semantic consistency (maximizing sim(vᵢ, vᵢ₋₁)). - Text editing instructions (Iₜ): Q appends mathematical transformations to the prior answer (e.g., "let Z = |A − 5|, find D = 3Z/4 + 9/4"), converting single-step problems into k-step reasoning chains. Ground-truth answers are computed via TTRL majority-vote sampling, reducing complexity from O(Nᵏ) to O(kN) via Markov factorization.

Adversarial training: Q is rewarded by a compound signal (Eq. 9): product of image semantic consistency scores × answer confidence scores, divided by the answer model's accuracy on the harder question. This incentivizes Q to produce questions that are difficult for A but still solvable. A is trained with binary correctness reward. The ratio of training steps follows a GAN-style schedule (multiple A updates per Q update).

Curriculum scaling: The instruction chain length k increases monotonically with training round (k = ⌊t/c⌋), guiding the model from k-step to (k+1)-step reasoning incrementally, providing implicit prefix-style process rewards.

Key Contributions

  • Progressive multi-modal instruction evolution framework that self-generates increasingly hard questions without external teacher models or human annotation.
  • Adversarial RL training loop between question generator Q and answer model A, using a composite reward that balances visual consistency, answer confidence, and question difficulty.
  • Curriculum-based instruction scaling that provides process-level intermediate rewards by training on subproblem chains before full k-step questions.
  • TTRL-based ground-truth derivation for evolved multi-step questions with reduced sampling complexity.

Results

  • MathVista-mini: 74.0 vs. 73.0 (MM-EUREKA), 72.9 (NoisyRollout), 67.8 (base Qwen2.5-VL-7B)
  • MathVision-full: 30.8 vs. 28.9 (NoisyRollout), 26.9 (MM-EUREKA), 24.7 (base); highest among all 7B open-source methods
  • MathVerse-mini: 51.8 vs. 52.8 (NoisyRollout), 50.3 (MM-EUREKA)
  • GSM8K: 85.1 vs. 84.1 (MindGYM), 83.6 (base)
  • MATH500: 73.2 vs. 68.4 (GRPO/NoisyRollout), 67.4 (base)
  • Average across 5 benchmarks: 63.0 vs. 63.9 (GPT-4o), 60.4 (NoisyRollout), 59.9 (MM-EUREKA)
  • General benchmarks (MMMU/MMStar/AI2D): 53.0 / 62.7 / 83.3 vs. 52.3/64.1/81.4 (MM-EUREKA)
  • Ablation (relative improvement): Base 100% → SFT +4.9% → GRPO +6.6% → ProgressiveGRPO +8.9% → EvolvedGRPO +12.1%

Limitations

  • Image editing relies on an external diffusion model (Flux.Kontext), introducing a non-trivial dependency and potential latency/cost at data-generation time.
  • Text editing instructions are restricted to mathematical transformations of numerical answers; applicability to open-ended or non-numeric VQA tasks is unclear.
  • Evaluation uses GPT-4o as the parsing judge, introducing variance relative to original benchmark reported numbers.
  • On challenging perception-heavy benchmarks (MMMU), all methods including EvolvedGRPO remain far below GPT-4o, indicating the approach does not fully bridge fine-grained visual perception gaps.
  • The adversarial training stability analysis is relegated to appendices; the GAN-style training dynamic may be sensitive to hyperparameters (curriculum step c, T, µ).

Relevance to Vision-Language Models

EvolvedGRPO directly targets a core failure mode of RL-based LVLM training: reward signal collapse due to visual-textual modality entanglement, which standard GRPO (designed for pure-text LLMs) does not handle. The self-evolving data generation paradigm sidesteps the upper-bound limitations of distillation-based approaches, which are dominant in the current LVLM reasoning literature, and offers a principled path to continually harder training data without human annotation. The dual-modality instruction evolution (image + text) and the prefix-style process reward via subproblem chains are practically useful patterns for the broader challenge of training multimodal reasoning models. This work is directly relevant alongside MM-EUREKA, NoisyRollout, and VLM-R1 as a key entry in the emerging landscape of RL-for-LVLM-reasoning.