Skip to content

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs

🕒 Published (v1): 2026-04-27 03:23 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Soft modality routing in MoE-VLMs; first study of modality-specific expert allocation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SMoES introduces a training-time regularization framework for Mixture-of-Experts VLMs that replaces hand-crafted or modality-agnostic routing with soft, layer-adaptive modality scores combined with an inter-bin mutual information objective. Applied to four MoE-VLM backbones across 16 benchmarks, it yields +0.9% multimodal and +4.2% language-only average gains over soft-routing baselines, while cutting expert-parallel (EP) communication overhead by 56.1% and improving throughput 12.3%.

Problem

Existing MoE-VLM routing strategies are either hand-crafted (hard/hybrid routing) or modality-agnostic (vanilla soft routing). Hard routing achieves high specialization but degrades performance severely (-4.4% multimodal, -22.2% language-only) because it ignores layer-varying modality fusion dynamics. Soft routing avoids rigid priors but provides no guidance for modality specialization, leaving expert capacity underutilized and creating inference inefficiency under expert parallelism due to modality-agnostic token scattering across devices.

Method

SMoES has three interlocking components:

Soft Modality Scores — tokens receive continuous scores \(M_{ij,m}^{(l)} \in [0,1]\) (summing to 1 over modalities \(m\)) per layer \(l\), computed by two estimators: - Attention-accumulated: initialized with hard binary modality IDs at layer 0, then propagated layer-by-layer via attention weights and updated with residual-norm weighting: $\(M_{ij,m}^{\text{attn},(l+1)} = \frac{\|\mathbf{x}_{\text{attn},ij}^{(l)}\| \cdot \tilde{M}_{ij,m}^{(l)} + \|\mathbf{x}_{ij}^{(l)}\| \cdot M_{ij,m}^{(l)}}{\|\mathbf{x}_{\text{attn},ij}^{(l)}\| + \|\mathbf{x}_{ij}^{(l)}\|}\)$ - Gaussian-statistics: maintains per-layer per-modality Gaussian distributions (diagonal covariance) updated via EMA-Welford; per-token soft scores derive from temperature-scaled log-likelihoods under each distribution.

Expert Binning — experts are sorted by a text-bias score \(f_{\text{spec}}(e) = \bar{C}_{\text{text},e}/(\bar{C}_{\text{text},e}+\bar{C}_{\text{vision},e})\) (EMA-tracked token counts) and partitioned into \(N_{\text{bins}}\) consecutive groups. Bins with similar modality preferences can be co-located on the same device under EP, reducing All-to-All communication.

Inter-Bin MI Regularization — maximizes \(I(M;B)\) between soft modality scores \(M\) and bin assignments \(B\): $\(\mathcal{L}_{\text{MI}} = -\sum_l \frac{1}{N_{\text{batch}}} \sum_i I_i(M; \mathbf{B})\)$ where joint probability \(P_i(m, \mathbf{B}_k)\) is computed from gating-score-weighted soft modality scores averaged over each bin. The full loss is \(\mathcal{L} = \mathcal{L}_{\text{task}} + \alpha_{\text{bal}}\mathcal{L}_{\text{bal}} + \alpha_{\text{MI}}\mathcal{L}_{\text{MI}}\) with bin-level load balancing.

Key Contributions

  • Two complementary soft modality score estimators (attention-accumulated and Gaussian-statistics) that capture layer-dependent, token-level modality fusion dynamics without hard boundaries.
  • Expert binning mechanism aligned with EP device-placement granularity, enabling modality-aware co-location of expert groups to reduce cross-device communication.
  • Inter-bin MI regularization objective that drives coherent modality specialization while remaining compatible with load-balancing constraints (unlike KL-based alternatives such as SMAR).
  • New Modality Specialization Index (MSI) metric quantifying per-expert deviation from uniform modality distribution across layers.
  • Extensive evaluation over four MoE-VLM backbones (DeepSeekMoE, OLMoE, Moonlight-MoE, Qwen3-MoE) and 16 benchmarks.

Results

  • Overall: +2.2% average over soft-routing baseline across all four backbones; +0.9% multimodal, +4.2% language-only.
  • DeepSeekMoE backbone: attention-soft SMoES +1.8% multimodal, +6.2% language; gaussian-soft +1.3% multimodal, +4.2% language, vs. no-specialization baseline (100%).
  • OLMoE backbone: attention-soft +0.5% multimodal, +6.7% language; gaussian-soft +0.6% multimodal, +4.3% language.
  • vs. Hard Routing: hard routing achieves MSI ≈ 1.0 but loses -4.4% multimodal and -22.2% language-only; SMoES achieves MSI 0.44–0.62 while strictly improving both.
  • vs. SMAR (best KL threshold): SMAR incurs -1.0% multimodal and -15.1% language-only (and requires disabling load-balancing); SMoES avoids both penalties.
  • vs. MoIIE (hybrid routing): MoIIE best variant still underperforms soft routing by ~3.1% multimodal and ~17.4% language.
  • Efficiency: 56.1% reduction in EP communication overhead; 12.3% throughput improvement under realistic deployment.
  • Ablation: KL inter-bin regularization (instead of MI) degrades by -1.5% multimodal and -8.5% language; adaptive binning outperforms fixed binning; hard-score (binary) modality input hurts multimodal performance (-0.8%) despite near-perfect MSI (0.904).

Limitations

  • Soft modality scores add per-layer computational overhead (attention accumulation or Gaussian distribution maintenance and log-likelihood computation across \(D\) dimensions per token).
  • Gaussian-statistics estimator requires sufficient batch statistics to form reliable per-layer distributions; behavior at very low batch sizes or under domain shift is not analyzed.
  • Expert binning with \(N_{\text{bins}}\) fixed to device count may not generalize gracefully to heterogeneous or dynamic cluster configurations.
  • Experiments use LLaVA-style two-stage training on fixed datasets (Pretrain-558K, Instruct-665K); generalization to instruction-tuning at larger scale or with RLHF/DPO is not studied.
  • The MSI metric is proposed but not validated against human judgments of specialization quality; a model can achieve moderate MSI without interpretable expert behavior.

Relevance to Vision-Language Models

SMoES directly addresses the efficiency-effectiveness tension in MoE-VLMs — the dominant architecture for frontier VLMs like DeepSeek-VL2, Kimi-VL, and InternVL — by providing a principled, data-driven alternative to ad-hoc routing heuristics. The finding that hard modality separation catastrophically degrades language capability is a concrete warning for MoE-VLM design, and the insight that layer-dependent fusion heterogeneity requires adaptive (not fixed) routing signals is a non-trivial empirical contribution. The EP communication reduction (56.1%) is practically significant given that MoE-VLMs are deployed under expert parallelism at scale. This work belongs to a growing line of research on controlling expert specialization in multimodal MoE models (alongside SMAR, LTDR, MoIIE) and provides stronger baselines than the existing methods.