Skip to content

NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

NoisyRollout augments GRPO-based RL training of VLMs by mixing rollouts from clean and moderately distorted images, injecting perceptual diversity to improve policy exploration. A sigmoid-shaped noise annealing schedule reduces distortion strength over training to maintain stability. The method requires no extra training cost and achieves state-of-the-art performance across visual reasoning and perception benchmarks with only 2.1K training samples.

Problem

RL-tuned VLMs struggle with two compounding issues: (1) insufficient policy exploration during RL training—standard remedies like increasing decoding temperature add superficial variance without meaningful behavioral diversity; and (2) imperfect visual perception that corrupts downstream reasoning. Existing VLM RL methods simply port LLM techniques and ignore these perceptual challenges.

Method

NoisyRollout extends GRPO with a hybrid rollout strategy: for each training sample (I, q), the old policy generates n₁ rollouts from the clean image and n₂ rollouts from a distorted image Äš = T_{α_t}(I). All n₁+n₂ rollouts form a single group for reward baseline and normalized advantage computation, but the policy update conditions only on the clean image. This creates two learning signals: (1) positive trajectories from distorted inputs reveal alternative reasoning paths; (2) reward differences between clean/noisy rollouts act as implicit contrastive signals that refine perception.

Distortion strength follows a sigmoid annealing schedule α_t = α₀ · (1 − 1/(1 + e^{−λ(t−γ)/t_max})), which starts high (encouraging exploration) and decays toward zero (on-policy convergence). Default distortion is Gaussian noise; rotation also validated. No changes to the GRPO objective or additional parameters beyond distortion hyperparameters.

Key Contributions

  • Hybrid rollout strategy that mixes clean and noisy trajectories within a single GRPO group without extra compute cost
  • Sigmoid noise annealing schedule preventing distributional mismatch from fixed-strength distortions
  • Quantitative gradient projection analysis showing noisy rollouts contribute disproportionately to policy updates, especially early in training
  • Demonstration that NoisyRollout provides complementary gains atop GRPO variants (no-std(r), higher clip Δ)
  • State-of-the-art open-source RL-tuned VLM results from only 2.1K geometry samples

Results

  • NoisyRollout-7B (Geometry3K, 2.1K) vs. vanilla GRPO: MathVerse 53.2% (+2.4%), MathVision 28.5% (+1.2%), MathVista 72.6% (+2.1%), WeMath 69.6% (+2.2%), HallusionBench 72.1% (+2.3%); avg. 59.2% vs. 57.2%
  • Outperforms OpenVLThinker-7B (35K SFT + 15K RL) on MathVerse (53.2% vs. 48.0%) and MathVista (72.6% vs. 71.5%)
  • Surpasses GPT-4o on MathVerse (53.2% vs. 50.8%) and WeMath (69.6% vs. 69.0%)
  • 32B model (Geometry3K): MathVerse 58.9% (+0.0%), MathVision 39.9% (+0.7%), MathVista 77.8% (+0.8%), HallusionBench 73.5% (+1.2%) over vanilla GRPO; smaller gains attributed to 32B base already RL-tuned
  • Scales consistently from 1.1K to 6.4K samples on MMK12; performance gains do not diminish with data scale
  • Rotation augmentation also outperforms GRPO (avg. 58.2% vs. 57.2%), confirming augmentation-type generality
  • Disabling noise annealing causes in-domain collapse at step ~45 and lowers OOD avg. to 58.0% vs. 59.2%
  • NoisyRollout with temperature 1.0 outperforms all vanilla GRPO temperature settings (0.8–1.4) and mixed-temperature variants

Limitations

  • Gains are smaller for the 32B model, likely because Qwen2.5-VL-32B-Instruct was already RL post-trained, leaving less room for exploration-driven improvement
  • Excessive noise (α₀ ≄ 600 Gaussian steps) causes divergence; the method requires careful noise calibration
  • Evaluated only on math-focused and geometric reasoning benchmarks; generalization to broader multimodal tasks (e.g., video, fine-grained VQA) is not demonstrated
  • Cropping and some other augmentation types were found ineffective (details deferred to appendix), suggesting not all distortions are suitable
  • Vision encoder is frozen during training for stability, which may limit perceptual adaptation capacity

Relevance to Vision-Language Models

NoisyRollout directly targets the RL fine-tuning regime that has become central to VLM reasoning advancement (GRPO, DeepSeek-R1-style training), addressing the underexplored axis of perceptual robustness during policy optimization. The key insight—that distorted-image rollouts provide contrastive perceptual signals and exploration diversity without modifying the RL objective—is immediately applicable to any GRPO-based VLM training pipeline. This work complements the broader trend of scaling test-time compute in VLMs by showing that input-side diversity, not just output-side temperature tuning, is a meaningful lever. The gradient projection analysis offers a principled diagnostic for understanding how augmented data contributes to VLM policy updates, which is valuable for future work on RL training dynamics.