Skip to content

PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PeRL addresses the failure of existing RL-trained VLMs on multi-image tasks by augmenting GRPO training with image-sequence permutation and a rollout filtering mechanism. The permutation diversifies positional contexts while correspondingly rephrasing answers to maintain semantic consistency. Qwen-PeRL (7B) achieves state-of-the-art on multi-image benchmarks while preserving single-image reasoning performance.

Problem

Existing RL-enhanced VLMs (DeepSeek-R1-style) are trained and evaluated almost exclusively on single-image tasks. When confronted with interleaved multi-image inputs, these models exhibit positional bias: shuffling the order of input images causes incorrect or inconsistent predictions, revealing that models conflate image identity with token position rather than learning order-invariant cross-image reasoning. Additionally, naive multi-image RL training suffers from severe difficulty imbalance in training data (mean accuracy 0.78 before filtering), causing unstable optimization.

Method

PeRL augments GRPO with two components, integrated as a plug-and-play module:

  1. Permutation GRPO: For each multi-image training sample \((x_1, y_1)\), \(n_s\) additional samples \(\{(x_i, y_i)\}\) are generated by randomly permuting the image sequence. A GPT-4o-based semantic equivalence check determines whether the answer \(y\) is invariant to permutation (\(S=1\)) or must be transformed via \(\Lambda(y, \sigma)\) (e.g., reindexing answer choices). A linearly decaying factor \(\alpha_t\) controls permutation probability over training. All \(n_s+1\) groups are merged; the advantage baseline \(\bar{R}\) is computed across all rollouts jointly, and the GRPO policy loss is conditioned on the permuted inputs \(x_i\) (not the original), avoiding the collapse that would arise from conditioning on original prompts with permuted answers.

  2. Rollout Filtering: Before RL training, Qwen2.5-VL-7B performs 10 rollouts per sample and computes average accuracy as a difficulty score. Samples are filtered to balance the difficulty distribution (mean shifts from 0.78 to 0.38), focusing learning on medium-difficulty trajectories.

Data pipeline: 22K multi-image samples curated from Mantis-Instruct (721K → 22K after rule-based filter + format rephrasing via GPT-4o + rollout filter + semantic variation check) combined with 36K single-image K12 math examples. Base model: Qwen2.5-VL-7B-Instruct; \(n_s=1\), \(n=6\) rollouts per order (12 total), trained 2 epochs, lr \(1\times10^{-6}\), batch 256, KL coefficient \(\beta=0.01\).

Key Contributions

  • Permutation-augmented GRPO that diversifies image-order exposure during RL rollouts and correctly re-conditions policy loss on permuted inputs.
  • Multi-stage data preprocessing pipeline: rule-based filter → GPT-4o format rephrasing → rollout-based difficulty filtering → GPT-4o semantic variation labeling.
  • Theoretical and empirical demonstration that permutation increases rollout diversity (measured via BGE-Large embedding cosine similarity) and mitigates positional bias.
  • State-of-the-art on 5 multi-image benchmarks with only 58K total training samples; competitive on 3 single-image math benchmarks.

Results

  • Mantis-Eval: 76.39 (+5.59 over Qwen2.5-VL-7B baseline 70.80; best among all 7B RL models, beating Noisy-K12 at 73.15)
  • BLINK: 58.53 (+3.30 over baseline; best RL model)
  • MMIU: 54.23 (+2.23 over baseline)
  • Remi (OOD): 43.38 (+7.00 over baseline; matches MM-Eureka 43.31)
  • MV-MATH (OOD): 25.68 (+5.27 over baseline)
  • MathVista: 73.00 (matches MM-Eureka 73.00, competitive with Noisy-K12 72.90)
  • MathVerse: 49.56 (below Noisy-K12 52.80 but above base 46.30)
  • Overall AVG (8 benchmarks): 51.13 vs. next best Noisy-K12 50.05
  • Ablation: \(n_s=2\) marginally outperforms \(n_s=1\) on multi-image tasks (Mantis-Eval 77.78, BLINK 59.50) but degrades MathVista/Remi and increases cost.

Limitations

  • Permutation augmentation introduces a trade-off: higher \(n_s\) improves spatial generalization but hurts reasoning benchmarks (MathVista, Remi) and reduces KV-cache reuse efficiency.
  • GPT-4o is required for semantic variation labeling and format rephrasing, making the data pipeline dependent on a proprietary model.
  • Training data is restricted to Mantis-Instruct + K12; generalization to other multi-image domains (video, 3D, document) is untested.
  • Permutation is applied uniformly with a linear decay schedule; the paper notes adaptive permutation strategies (per-difficulty, per-uncertainty) as future work.
  • Single-image performance on MathVerse (49.56) falls behind Noisy-K12 (52.80), indicating the multi-image training focus incurs a cost on some reasoning tasks.

Relevance to Vision-Language Models

PeRL directly addresses a well-known but underexplored failure mode of RL-trained VLMs: positional bias in multi-image interleaved reasoning. It extends the DeepSeek-R1/GRPO paradigm—dominant in single-image VLM reasoning—to the more complex and practically important multi-image setting without architectural changes, making it a straightforward drop-in extension for any GRPO-based VLM training recipe. The rollout filtering strategy is broadly applicable as a data efficiency technique for RL fine-tuning of VLMs. The result that multi-image RL training improves OOD math reasoning (Remi, MV-MATH) without explicit supervision on those tasks suggests that robust cross-image relational reasoning transfers to compositional reasoning more generally, with implications for scaling interleaved VLM training.