Skip to content

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes UnifiedGRPO, a two-stage post-training strategy that enables unified vision-language models to generate interleaved text-image sequences without large-scale interleaved training data. It extends GRPO to the multimodal setting by treating text and image token generation as a single autoregressive decision trajectory, optimized with hybrid rewards and process-level supervision.

Problem

Most unified VLMs (Show-O, VILA-U, ILLUME) produce only single-modality outputs at inference time despite being trained on both text and images, because they lack fine-grained supervision for dynamic modality transitions. Existing RL alignment methods (RLHF, GRPO) apply only to text-only policies, and outcome-level reward signals are too sparse for the complex interleaved generation setting.

Method

Stage 1 — Warm-up: Fine-tune a pretrained unified model (VILA-U) on a hybrid dataset: 0.3M interleaved text-image samples (ActivityNet, GenHowTo, OpenStory++) plus 1M multimodal understanding samples and 1M text-to-image samples. This exposes the model to modality-switching patterns while preventing catastrophic forgetting.

Stage 2 — GRPO fine-tuning: Extend GRPO to multimodal outputs by unifying text and image tokens under a single decoding trajectory. For each prompt, G=4 candidate responses are sampled; token-level advantages are computed by group-normalizing a hybrid scalar reward: - r_t (textual): relevance and coherence of generated text (rule/model-based scorer) - r_v (visual): image quality and image-text alignment via ImageReward - r_f (format): structural fidelity enforced via <think> / <vis> special tokens

Process-level rewards are additionally assigned at each modality-step boundary, with token-level advantages accumulated as the sum of all future step rewards — providing denser, step-wise learning signal vs. end-of-sequence outcome rewards only.

Key Contributions

  • Two-stage post-training pipeline that activates latent interleaved generation in pretrained unified models using minimal interleaved data (0.3M warm-up, 0.1M GRPO)
  • Unified GRPO objective that jointly models text and image generation under one autoregressive trajectory with a single KL-divergence term across modalities
  • Hybrid reward combining textual relevance, visual quality/alignment, and format fidelity
  • Process-level reward that assigns intermediate rewards at each modality-switching step, improving training efficiency and coherence
  • QLoRA variant achieving competitive performance at reduced compute

Results

  • MMIE (20K queries): Ours 59.50% AVG vs. Anole 55.22%, GILL 51.58%, MiniGPT-5 50.92%, EMU-2 45.33% (all at 7B except EMU-2 at 37B)
  • InterleavedBench (815 instances): Ours 3.13 AVG vs. GILL 1.84, MiniGPT-5 1.82, EMU-2 1.68
  • Ablation: warm-up alone achieves 53.31% MMIE / 1.97 InterleavedBench; adding GRPO raises to 59.50% / 3.13 (+6.19% / +1.16)
  • Ablation: progressive reward addition — r_f only → 53.56 / 2.05; +r_t → 54.62 / 2.30; +r_v → 57.83 / 2.79; +process reward → 59.50 / 3.13
  • G=4 vs. G=2 generations: +4.36% MMIE, +0.86 InterleavedBench
  • ImageReward outperforms CLIP-score as visual reward (59.50 vs. 56.94 on MMIE)
  • Standard understanding benchmarks (MME-P 1425.2, MMvet 32.8) remain comparable to VILA-U baseline, confirming no catastrophic forgetting

Limitations

  • No notable improvement on general multimodal understanding or text-to-image generation tasks; GRPO mainly aligns modalities rather than improving base capabilities
  • Performance ceiling determined by the base model (VILA-U); stronger unified architectures needed to push further
  • GRPO is computationally expensive in the multimodal setting (requires generating visual tokens per rollout); G beyond 4 was not explored due to GPU constraints
  • Image resolution fixed at 256×256, which may limit visual fidelity
  • Reward models (text scorer, ImageReward) may introduce their own biases

Relevance to Vision-Language Models

This work directly addresses the gap between unified VLMs that understand and generate in isolation versus models that can produce tightly coupled, alternating text-image sequences — a prerequisite for visual storytelling, step-by-step visual reasoning, and multimodal dialogue. The extension of GRPO to a joint text+image policy is a concrete RL alignment recipe that any unified autoregressive VLM (Chameleon-style token-interleaved architectures) can adopt as a post-training stage. The hybrid reward design (textual + visual + format + process-level) provides a reusable scaffold for training reward signals in multimodal RL settings where outcome-level signals are insufficient.