Skip to content

MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models

🕒 Published (v1): 2026-03-26 03:23 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; GRPO-based reinforcement learning improves MoE routing in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoE-GRPO replaces the deterministic top-\(K\) expert routing in Mixture-of-Experts VLMs with a reinforcement learning framework built on Group Relative Policy Optimization (GRPO), treating expert selection at each layer as a sequential decision-making problem. By jointly optimizing token-level generation (Token-GRPO) and layer-wise gating (Gate-GRPO), together with a modality-aware router guidance that prunes infrequently-activated experts per modality, MoE-GRPO achieves consistent accuracy gains over deterministic and stochastic routing baselines across image and video benchmarks.

Problem

Standard MoE architectures select experts via a deterministic top-\(K\) mechanism over gating scores \(g^l(h_{t,l}) \in \mathbb{R}^N\), which (1) restricts exploration of diverse expert combinations, (2) causes expert overfitting to a small subset, and (3) provides no explicit optimization signal for the routing policy itself. Heuristic stochastic alternatives (Gaussian noise, multinomial sampling) add exploration but do not optimize the routing policy, so suboptimal expert combinations persist.

Method

MoE-GRPO formulates expert selection as a sequential decision problem and applies GRPO to optimize it. The action space is extended from the output token sequence to include expert routing decisions across all tokens and layers: \([o_{1,1}, o_{1,2}, \ldots, o_{T,L}]\).

Token-GRPO optimizes token-level generation quality using the standard GRPO clipped objective conditioned on sampled expert rollouts \(\{E^i\}_{i=1}^G \sim g_\text{old}(E|x)\):

\[\mathcal{L}_\text{Token-GRPO} = \mathbb{E}\left[-\frac{1}{|y^i|}\sum_{t=1}^{|y^i|} \min\left[r_t^i \hat{A}^i, \text{clip}(r_t^i, 1-\epsilon, 1+\epsilon)\hat{A}^i\right]\right]\]

where \(r_t^i = \pi_\theta(y_t^i | x, y_{<t}^i; E_{<t}^i) / \pi_\text{old}(y_t^i | x, y_{<t}^i; E_{<t}^i)\).

Gate-GRPO directly optimizes each gating network \(g^l\) using token- and layer-level importance ratios \(\hat{r}_{t,l}^i = g^l_\theta(E_{t,l}^i | \cdot) / g^l_\text{old}(E_{t,l}^i | \cdot)\), propagating the group-relative advantage \(\hat{A}^i\) to every layer, yielding dense supervision for routing.

The final objective is \(\mathcal{L}_\text{MoE-GRPO} = \mathcal{L}_\text{Token-GRPO} + \mathcal{L}_\text{Gate-GRPO}\). No KL regularization toward a reference policy is used since gating networks are trained from scratch.

Modality-aware router guidance precomputes normalized vision- and text-awareness scores \(\hat{s}_v(e_i)\) and \(\hat{s}_t(e_i)\) from top-\(K\) activation frequencies, then masks the bottom 25% of modality-mismatched experts (gating score → \(-\infty\)) before multinomial rollout sampling, reducing wasteful cross-modality exploration.

Experiments use InternVL3.5-1B converted to MoE (\(K=2\), \(N=8\), 1.3B activated / 2.9B total), trained on 100K visual instruction examples for 25K steps; accuracy-based binary reward (correct = 1, incorrect = 0).

Key Contributions

  • First formulation of MoE expert selection as a sequential decision-making problem optimized via RL (GRPO).
  • Dual-objective MoE-GRPO loss combining Token-GRPO (output-level) and Gate-GRPO (layer-wise gating), providing dense routing supervision without a reference policy.
  • Modality-aware router guidance that constrains rollout exploration to modality-relevant experts via precomputed activation frequency scores, improving training stability and efficiency.
  • Empirical demonstration of task-level expert specialization, diverse expert utilization, and improved cross-dataset and domain generalization.

Results

  • Multi-modal understanding (Tab. 1): MoE-GRPO (1.3B activated) achieves 56.0% average across 9 benchmarks vs. Det-FT 54.0% (+2.0%), Stoch-FT-Multi 53.7% (+2.3%), Stoch-FT-Noise 54.3% (+1.7%); outperforms on 7/9 benchmarks.
  • Notable per-benchmark gains over Det-FT: MMBench 77.5 vs. 75.8, MMStar 45.7 vs. 45.6, MMT-Bench 65.8 vs. 62.7, VideoMME 46.6 vs. 45.6, MLVU 53.1 vs. 48.6, LongVideoBench 46.5 vs. 45.3, MVBench 58.4 vs. 56.7.
  • Cross-dataset generalization (Tab. 2): CLIP-MoE + MoE-GRPO achieves 71.9% average across 10 target datasets vs. CLIP-MoE + Det-FT 68.8% (+3.1%); Det-FT degrades below the base CLIP-MoE (70.8%).
  • Domain generalization (Tab. 3): MoE-GRPO 67.5% average vs. Det-FT 66.0% (+1.5%) and CLIP-MoE 63.4% (+4.1%) across 5 ImageNet variants.
  • Ablation (Tab. 4): Gate-GRPO alone collapses to 50.9% avg; Token-GRPO alone achieves 53.9%; both together reach 55.7% (full model on the ablation subset).
  • Modality-aware guidance (Tab. 5): Modality-aware guidance 55.7% avg vs. modality-agnostic multinomial 54.8% and agnostic noise 54.2%.

Limitations

  • Evaluated only on a 1B-parameter base model (InternVL3.5-1B → 2.9B MoE); scalability to larger MoE VLMs is unverified.
  • MoE architecture is manually constructed by converting a dense model; performance depends on that conversion quality and the fixed \(K=2\), \(N=8\) hyperparameters.
  • Binary accuracy reward is task-specific to multi-choice QA; extension to open-ended generation requires alternative reward design.
  • No KL regularization against a reference policy is used because gating networks start from scratch; this may affect training stability in other settings.
  • Inference reverts to deterministic top-\(K\), so runtime gains from stochastic exploration are training-only.
  • Cross-dataset experiments use only CLIP-MoE for the vision-encoder setting; integration into end-to-end VLM cross-dataset generalization is not evaluated.

Relevance to Vision-Language Models

MoE-GRPO directly addresses a bottleneck in scaling VLMs efficiently: the deterministic routing that wastes sparse model capacity through expert overfitting. By applying GRPO — already successful for improving VLM reasoning — to the internal routing mechanism rather than just the output, this work opens a new axis of RL-based optimization inside multimodal architectures. It connects the growing line of GRPO-for-VLMs work (e.g., Video-R1) with MoE architecture research, showing that reward-driven exploration can induce task-level expert specialization that improves zero-shot and cross-domain generalization. Researchers tracking VLMs should note this as an example of how RL fine-tuning can reach inside model internals, not just surface outputs.