Skip to content

Spotlight on Token Perception for Multimodal Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Token-level perceptual spotlight improves RLVR visual reasoning in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VPPO (Visually-Perceptive Policy Optimization) is a novel policy gradient algorithm for multimodal RLVR that explicitly incorporates visual token dependency into the learning signal. It addresses the finding that standard GRPO/DAPO broadcast a uniform reward across all tokens and trajectories, failing to distinguish visually-grounded reasoning from linguistic shortcuts. On eight multimodal benchmarks, VPPO (7B) achieves 57.5% average accuracy, a 19.2% improvement over the base Qwen2.5-VL-7B baseline.

Problem

Existing RLVR frameworks for LVLMs (e.g., GRPO, DAPO) apply a single coarse outcome reward uniformly to every token and trajectory. This creates two failure modes: (1) trajectory-level ambiguity — correct answers reached via linguistic priors or hallucination are rewarded equally to those grounded in visual evidence; (2) token-level uniformity — reward signal is diluted across perceptually irrelevant tokens, impeding efficient learning of visually-grounded reasoning.

Method

VPPO quantifies each token's visual dependency as the KL divergence between the model's output distribution conditioned on the real image \(I\) versus a perturbed (non-informative) image \(I'\):

\[S(s_t, I) := D_{\mathrm{KL}}\!\left(\pi_\theta(\cdot \mid s_t, I) \,\|\, \pi_\theta(\cdot \mid s_t, I')\right)\]

Two hierarchical control mechanisms are built on this score:

  • Token-level Gradient Filtering (TGF, micro): A binary mask \(m_{i,t} = \mathbf{1}(t \in K_i)\) retains gradients only for the top-\(k\%\) highest-dependency tokens (default \(k=0.4\)), suppressing noise from perceptually irrelevant tokens.
  • Trajectory-level Advantage Shaping (TAS, macro): Each trajectory's GRPO advantage is rescaled by a shaping factor \(\alpha(\tau_i)\) computed from batch-normalized trajectory dependency \(\bar{S}(\tau_i)\) (mean of token scores), mapped to \([\beta_{\min}, \beta_{\max}]\). Trajectories with high visual engagement receive amplified updates.

The combined VPPO objective is:

\[\mathcal{L}_{\mathrm{VPPO}}(\theta) = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} m_{i,t} \cdot \min\!\left(r_{i,t}(\theta)\hat{A}'_i,\, \mathrm{clip}(r_{i,t}(\theta), 1{-}\varepsilon, 1{+}\varepsilon)\hat{A}'_i\right)\right]\]

where \(\hat{A}'_i = \alpha(\tau_i) \cdot \hat{A}_{\mathrm{GRPO},i}\). The perturbed image forward pass is reused from the rollout to reduce compute overhead.

Key Contributions

  • First analysis of multimodal RLVR through the lens of token-level visual dependency, revealing two empirical findings: (1) token visual dependency is sparsely distributed (exponential decay), and (2) trajectories exhibit heterogeneous visual grounding.
  • VPPO: a plug-in policy gradient algorithm combining TGF and TAS that reshapes the GRPO/DAPO learning signal hierarchically using KL-divergence-based visual dependency.
  • Theoretical proof that VPPO reduces gradient estimator variance by a factor of approximately \(k \cdot \mathbb{E}[\alpha(\tau)^2] < 1\) relative to GRPO.
  • Demonstrated state-of-the-art results at both 7B and 32B scales with faster convergence, without any supervised fine-tuning pre-step.

Results

  • VPPO-7B: 57.5% avg across 8 benchmarks vs. 55.0% DAPO-7B, 55.3% PAPO-D-7B (best prior); +19.2% over base Qwen2.5-VL-7B (38.3%).
  • VPPO-32B: 64.6% avg vs. 63.5% DAPO-32B, 62.6% GRPO-32B; surpasses MM-Eureka-32B (60.8%) and NoisyRollout-32B (58.1%).
  • Ablation: TGF alone (+2.1% over DAPO) > TAS alone (+1.3%); combined VPPO (+2.5%) is best, confirming synergy.
  • Optimal gradient filtering ratio \(k=0.4\); too-low \(k\) starves the signal, too-high \(k\) reintroduces noise.
  • Best TAS config: \(\beta_{\min}=0.9\), \(\beta_{\max}=\text{Dyn.}\) (batch-adaptive upper bound).
  • Benchmarks: MathVerse, DynaMath, MMK12, Geo3k, MathVision, We-Math, LogicVista, MMMU-Pro (all exact-match, avg@8).

Limitations

  • The perturbed-image forward pass adds computational overhead per training step, even if partially amortized by policy reuse.
  • The top-\(k\%\) threshold and \([\beta_{\min}, \beta_{\max}]\) range are hyperparameters requiring tuning; the dynamic \(\beta_{\max}\) scheme is batch-dependent and may be sensitive to batch composition.
  • Theoretical variance reduction result (Theorem 3.1) relies on assumptions detailed in the appendix and is approximate.
  • Evaluated only on Qwen2.5-VL backbone; generalizability to other LVLM families (e.g., LLaVA, InternVL) is not demonstrated in the main paper.
  • The random masking perturbation as the non-informative \(I'\) may not fully eliminate visual information (residual texture/color), which could affect dependency measurement fidelity.

Relevance to Vision-Language Models

VPPO directly addresses a fundamental misalignment in how RL training signals are applied to VLMs: the inability of standard RLVR to distinguish visually-grounded reasoning from language-prior shortcuts. By introducing a token-level perceptual dependency score derived from KL divergence between masked and unmasked image conditions, this work provides a principled diagnostic tool and optimization primitive for multimodal reasoning VLMs. It is directly relevant to the line of work on RLVR for LVLMs (GRPO/DAPO extensions) and complements reward-centric and data-centric approaches by intervening at the gradient level. The sparse token perception finding also has implications for efficient inference and attention analysis in VLMs more broadly.