Skip to content

SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

🕒 Published (v1): 2026-07-15 15:13 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SIVA-RL addresses a core failure mode of visual RL for VLMs: existing perturbation-based methods assume the type of visual intervention determines the training signal, but the same operator produces heterogeneous behavioral outcomes across samples. SIVA-RL replaces operator-conditioned regularization with sample-wise, outcome-conditioned routing that uses the observed reward drop to direct each clean–intervention pair toward sensitivity alignment, invariance alignment, or abstention. Applied to GRPO and DAPO backbones at 3B and 7B scale, it consistently outperforms matched RL baselines across nine multimodal reasoning benchmarks.

Problem

RLVR trains VLMs on answer-level correctness, which does not ensure the model grounds predictions in visual evidence — a model can earn high reward via language priors, frequency biases, or memorized templates. Existing visual-intervention methods (e.g., PAPO) contrast policy behavior on original vs. modified images, but assign supervision by intervention type rather than observed effect. An empirical audit of 906 clean-correct examples shows that black-image, random-mask, and PatchSwap interventions each produce outcome-drop rates of ≈62–68% and outcome-stable rates of ≈32–38%, meaning the same operator produces mixed outcomes within a single clean-correct subset and cannot prescribe a uniform optimization direction.

Method

SIVA-RL augments a standard RLVR backbone (GRPO or DAPO) with three components:

1. Localized PatchSwap construction. The image is partitioned into a token-aligned grid \(\Omega_i\). Target cells are sampled with probability \(\eta\) (eligible) and \(\rho\) (active). Each active target cell \(u\) is replaced with a distant source cell \(s_i(u)\) satisfying \(\bar{d}_i(u, v) \geq d_{\min}\), producing intervention \(\tilde{x}_i\) that disrupts local evidence and spatial structure while preserving global image statistics.

2. Outcome-conditioned soft routing. A frozen audit policy \(\pi_{\bar{\theta}}\) decodes \(M\) responses under both views and computes the reward drop \(\Delta_i = R_i^{cl} - R_i^{ci}\). A clean-correctness gate \(b_i = \mathbf{1}[R_i^{cl} \geq \tau_c]\) filters pairs. Soft routing weights are: $\(w_i^{sen} = b_i \cdot \text{Ramp}(\Delta_i;\, \delta_{sen}^{lo}, \delta_{sen}^{hi}), \quad w_i^{inv} = b_i \cdot \left(1 - \text{Ramp}(\Delta_i;\, \delta_{inv}^{lo}, \delta_{inv}^{hi})\right)\)$ with a gap between thresholds forming an abstention band for ambiguous pairs.

3. Clean-anchored dual-direction alignment. Existing clean rollouts are re-scored under both visual conditions without additional sampling. A stop-gradient anchor on the clean log-likelihood defines a token-level KL surrogate \(d_{i,g,t} = \exp(z_{i,g,t}) - z_{i,g,t} - 1\) where \(z_{i,g,t} = \ell^{ci}_{i,g,t} - \text{sg}[\ell^{cl}_{i,g,t}]\). Two margin losses are applied: $\(L_{sen} = \frac{\sum_{i,g} w_i^{sen} [m_{sen} - D_{i,g}]_+}{\sum w_i^{sen}}, \quad L_{inv} = \frac{\sum_{i,g} w_i^{inv} [D_{i,g} - m_{inv}]_+}{\sum w_i^{inv}}\)$ The full objective is \(L_{SIVA} = L_{base} + \lambda_{sen} L_{sen} + \lambda_{inv} L_{inv}\).

Key Contributions

  • Operator–effect heterogeneity audit: Empirically demonstrates that intervention identity cannot determine optimization direction; outcome-drop rates for identical operators vary substantially across samples.
  • Outcome-conditioned selective supervision: Formalizes visual RL augmentation as a routing problem driven by the empirical reward drop \(\Delta_i\), decoupling intervention construction from supervision assignment.
  • SIVA-RL framework: Token-aligned distance-constrained PatchSwap + frozen paired outcome valuation + clean-rollout re-scoring + margin-bounded dual-direction alignment — compatible with both GRPO and DAPO without additional stochastic rollouts beyond the audit.

Results

  • Overall improvement over matched RL baselines (Table 1): relative gains of +6.83% (DAPO-3B), +7.01% (GRPO-7B), +12.46% (GRPO-3B), and +14.98% (DAPO-7B).
  • Vision-dependent subset gains: DAPO-7B achieves +8.79 pp on vision-dependent benchmarks vs. +7.81 pp on general; GRPO-3B achieves +7.63 pp vision-dependent vs. +4.63 pp general — showing larger gains precisely where visual grounding matters.
  • Counting benchmark: GRPO-3B jumps from 55.81 → 75.19 (+19.38 pp); GRPO-7B from 73.94 → 88.75 (+14.81 pp).
  • Comparison with 7B systems (Table 2): DAPO-7B + SIVA-RL achieves +14.98% relative overall; GRPO-7B + SIVA-RL outperforms PAPO-7B, DVRP-7B, and most listed math-specific VLMs on the full 6-benchmark math suite.
  • Ablation (Table 3): Distance-constrained PatchSwap + SIVA-RL yields +12.46% vs. random mask + SIVA-RL at +4.17% and PatchSwap without distance constraint + SIVA-RL at +6.68%, confirming both localization and distance constraint contribute.
  • Stepwise ablation (Table 4): Each component — PatchSwap (+5.09%), hard→soft sensitivity (+2.07%+2.90%), warm-up (+1.58%), invariance branch (+2.37%) — adds incrementally.

Limitations

  • The paired outcome change \(\Delta_i\) is used as an operational routing signal, not a certified measure of visual grounding; stable outcomes can still arise from language priors, redundant evidence, or task-irrelevant perturbations.
  • PatchSwap is within-image, so it cannot inject out-of-distribution visual content; its disruption is bounded by the image's own patch statistics.
  • The frozen audit policy is a no-gradient snapshot of the current policy, which may lag the evolving policy distribution during training.
  • Evaluation is limited to Qwen2.5-VL-3B and 7B backbones on ViRL39K training data; generalization to other architectures or training corpora is not demonstrated.
  • Medical-domain results are relegated to the appendix and not discussed in the main text, leaving domain transfer unclear.

Relevance to Vision-Language Models

SIVA-RL directly targets a known weakness of RLVR-trained VLMs: conflating linguistic reasoning with visual grounding. By diagnosing and correcting the operator-conditioned assumption baked into prior perturbation methods (e.g., PAPO), it advances the understanding of how auxiliary visual supervision should be assigned during RL fine-tuning of VLMs. The framework is backbone-agnostic (GRPO/DAPO) and scale-agnostic (3B/7B), making it directly applicable to current multimodal RL pipelines built on models like Qwen2.5-VL. The explicit separation of intervention construction from outcome valuation is a methodological contribution that generalizes beyond PatchSwap to any intervention strategy, positioning this work as a principled foundation for future visual-grounding objectives in VLM RL.