RL Forgets! Towards Continual Policy Optimization

🕒 Published (v1): 2026-07-05 15:46 UTC · Source: Arxiv · link

Why this paper was selected

Exposes catastrophic forgetting in RL-based VLM post-training — critical for continual adaptation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper challenges the prevailing belief that reinforcement learning (RL) is inherently resistant to catastrophic forgetting during continual VLM post-training, demonstrating empirically on a new benchmark (MRCL) that RL suffers just as severely as supervised fine-tuning. The authors propose Continual Policy Optimization (CPO), a replay-free framework that approximates the prior-task behavioral KL divergence via parameter-movement regularization. On Qwen3-VL-8B, CPO reduces forgetting by 13.7% and improves pretrained capability by 7.0%.

Problem

Existing evidence that RL resists catastrophic forgetting in continual VLM post-training is based on outdated or narrow benchmarks (datasets from 2018–2023 used with 2025 models, or single-task evaluations like jigsaw puzzles), creating potential contamination with pretraining corpora. No rigorous continual RL framework exists that avoids data replay while theoretically bounding prior-task performance degradation. The GRPO KL term \(\mathbb{E}_{x \sim T_\text{new}}[\mathrm{KL}(\pi_\theta \| \pi_\text{old})]\) is computed on current-task data and does not constrain behavioral drift on prior tasks.

Method

MRCL Benchmark. Five multimodal reasoning tasks from datasets released after mid-2025 (MedBookVQA, Navigation, We-Math2.0, Puzzle, FinMME), sequentially ordered to evaluate plasticity and stability under genuine distribution shift with minimal pretraining contamination.

Theoretical grounding. The paper proves (Proposition 3.1) that prior-task performance degradation is bounded by the token-level KL under the old policy's prefix visitation distribution: $\(|\eta_{i-1}(\tilde{\pi}) - \eta_{i-1}(\pi)| \leq B_{i-1}\sqrt{\frac{H}{2}\bar{D}^{d^{T_{i-1}}_{\tilde{\pi}}}_{\mathrm{KL}}(\tilde{\pi}, \pi)}\)$

Replay-free surrogate. Since the oracle KL requires old-task data, CPO performs a second-order Taylor expansion around the previous checkpoint \(\tilde{\theta}\), showing the Hessian equals the diagonal FIM. Proposition 3.2 then establishes that empirical parameter movement \(s_k^{(i-1)} = \tilde{\theta}_k - \theta_k^{(i-2)}\) is a noisy ranking proxy for FIM diagonal entries (expected squared movement \(\propto \eta^2 \sigma_A^2 C_k F_{kk}^{T_{i-1}}\)).

CPO algorithm. Maintains a cumulative protected set \(S^{(i-1)}\) of high-movement parameters. At each task \(T_i\), optimizes: $\(\mathcal{L}_i(\theta) = \mathcal{L}^\mathrm{RL}_i(\theta) + \frac{\lambda}{\|m^{(i-1)}\|_0} \left\| m^{(i-1)} \odot (\theta - \tilde{\theta}) \right\|_1\)$ After training, expands the protected set by adding the top \(p\%\) moving coordinates. L1 (not L2) regularization is used because the movement proxy yields binary rank information rather than cardinal Fisher magnitudes, encouraging sparse movement within the protected subset.

Key Contributions

• MRCL benchmark: five diverse multimodal tasks (medical VQA, spatial navigation, geometry, visual puzzles, financial charts) from post-mid-2025 datasets, designed to minimize pretraining contamination.
• Empirical falsification: demonstrates that both GRPO and GSPO suffer severe catastrophic forgetting on MRCL, disproving the "RL is immune" narrative.
• Theoretical forgetting bound: proves the prior-task KL under the old prefix visitation distribution bounds performance degradation (Proposition 3.1).
• CPO: replay-free, gradient-free continual RL method using masked sparse L1 movement regularization on a cumulatively accumulated set of important parameters.
• Evaluation across three model scales (Qwen3-VL-2B, 4B, 8B) with three aggregated metrics (MFT, MFN, MTA).

Results

• Qwen3-VL-8B: CPO reduces forgetting (CAD) by 13.7% and improves pretrained capability by 7.0% relative to baselines.
• CPO consistently achieves a better plasticity–stability Pareto frontier than GRPO, GSPO, LoRA, and Full SFT across all three model scales (2B, 4B, 8B) as shown in Fig. 1.
• Both RL methods (GRPO, GSPO) and SFT methods exhibit severe forgetting on MRCL, with large negative CAD values across all five sequential stages.
• CPO is built on top of GSPO and outperforms it on stability while maintaining comparable or better target-task accuracy improvement (CAI).

Limitations

• The parameter-movement proxy preserves ordinal rank but not cardinal Fisher magnitudes, so all protected parameters are treated as equally important by the binary mask.
• The protected set \(S^{(i)}\) grows cumulatively across tasks, which may become a bottleneck in very long task sequences.
• Experiments are limited to Qwen3-VL (2B/4B/8B); generalization to other VLM families is not shown.
• The theoretical bound (Proposition 3.1) assumes bounded rewards and local stationarity of SGD, which may not hold in all practical RL training regimes.
• The paper text is truncated before full results tables are presented, so quantitative comparisons on individual tasks are not available here.

Relevance to Vision-Language Models

This work directly addresses continual post-training of VLMs—a critical problem as models like Qwen3-VL are adapted to successive specialized domains. It corrects a widespread misconception in the VLM community that RL-based fine-tuning (GRPO, GSPO, DAPO) is inherently safer than SFT with respect to catastrophic forgetting. The MRCL benchmark, using post-2025 datasets, provides the first contamination-resistant testbed for evaluating forgetting in modern VLMs. CPO's approach—bridging behavioral KL theory with gradient-free parameter importance—offers a practical recipe for deploying VLMs in dynamic, privacy-sensitive environments where old data cannot be replayed.