RLAP-CLIP: Continual Multimodal Learning with Prototype Adaptation and Difficulty-Aware Routing¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; prototype adaptation with difficulty routing for class-incremental CLIP

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RLAP-CLIP tackles class-incremental learning with CLIP by replacing passive prototype averaging with a reinforcement-learning policy that actively weights exemplar samples for better class separability. It adds a difficulty-aware mixture-of-experts routing mechanism and balanced dual-modal (visual + textual) prompting. Across eight benchmarks, it achieves up to 4.52 pp average accuracy and 6.26 pp final accuracy gains over prior SOTA.

Problem¶

Two compounding failures limit CLIP-based continual learning: (1) prototype quality degrades progressively because class prototypes are computed as uniform averages of a small exemplar set — as the feature space shifts across tasks, these drift into suboptimal regions; (2) existing methods are text-centric, treating the visual encoder as a frozen feature extractor, which fails for fine-grained recognition tasks where discriminative patterns lack precise linguistic descriptors.

Method¶

RLAP-CLIP freezes both CLIP encoders and introduces three trainable components on top:

RLPO (Reinforcement Learning-based Prototype Optimization): Prototype construction is recast as a weighted summation \(p_c = \sum_{i:y_i=c} w_i f_i\) where a 4-layer MLP policy network \(\pi_\theta(w_i \mid f_i, \mathbf{P})\) predicts per-sample weights. The reward \(R_i = \text{sim}(f_i, p_{y_i}) - \max_{j \neq y_i} \text{sim}(f_i, p_j) - \lambda \sum_{j \neq y_i} \frac{\text{sim}(p_{y_i}, p_j)}{C-1}\) promotes intra-class cohesion and inter-class margin. Training uses a REINFORCE-style objective with a KL penalty against a reference policy \(\pi_\text{ref}\) (updated via EMA after each task) to prevent policy collapse.

Dual-Modal Prompting with Expandable Projections: Per-task learnable visual prompts \(V^t \in \mathbb{R}^{K \times D}\) are appended to image patch embeddings; learnable textual prompts \(T^t \in \mathbb{R}^{M \times D}\) are prepended to class templates. Task-specific linear projections \(P_v^t\), \(P_t^t\) map frozen encoder outputs to a task-adapted subspace \(\mathbb{R}^{D'}\), isolating task interference. Exemplars are refreshed via center-based selection: the \(m\) samples nearest to the updated class centroid are retained.

Difficulty-Aware Cross-Modal Fusion with MoE: A per-sample difficulty score \(d_i = 1 - \frac{\text{sim}(f_i^v, p_{y_i}^v) + \text{sim}(f_i^t, p_{y_i}^t)}{2}\) routes samples via \(P(E_\text{easy} \mid x_i) = \sigma(-\alpha(d_i - \tau))\) to either a single linear layer (easy) or a 3-layer FFN (hard), with soft mixing. Fused features (expert output + visual prototype + textual prototype) are re-weighted by a learned cross-modal attention before nearest-prototype classification.

Key Contributions¶

RLPO: reformulates prototype construction as an RL problem with a separability-aware reward, providing active optimization instead of passive averaging
Difficulty-aware MoE routing that allocates model capacity proportional to sample complexity, using RLPO-optimized prototypes as difficulty anchors
Enhanced dual-modal prompting showing that visual prompts add complementary benefit over text-only approaches (separation ratio 3.11 vs. 2.74 for text-only on CUB-200)
Center-based exemplar selection that keeps stored samples representative as the feature space evolves
Comprehensive evaluation across 8 datasets covering general, fine-grained, and domain-shift scenarios

Results¶

All comparisons use exemplar-replay baselines with 20 exemplars per class; RLAP-CLIP uses ViT-B/16 CLIP backbone:

CIFAR-100: 86.64% avg / 79.41% final — +3.72 avg over PROOF, +4.89 avg over C-CLIP
CUB-200: 85.78% avg / 83.67% final — +3.64 avg over C-CLIP (82.14%)
Cars: 94.82% avg / 93.15% final — +2.64 avg over C-CLIP (92.18%)
Aircraft: 70.25% avg / 68.41% final — +4.52 avg over C-CLIP (65.73%), the largest avg gain
UCF-101: 97.68% avg / 95.80% final — +5.83 avg over C-CLIP (91.85%)
ImageNet-R: 85.63% avg / 82.22% final — +2.48 avg over C-CLIP (83.15%)
ObjectNet: 53.89% avg / 48.91% final — +2.52 avg over C-CLIP (51.37%)
Food-101: 88.24% avg / 86.88% final — modest gain (+1.16 over C-CLIP)
Dual-modal prompting achieves 63.7% higher separation ratio vs. no prompting on CUB-200 (1.90→3.11); ablation shows RLPO adds the single largest per-step gain on Aircraft (+2.28 pp)

Limitations¶

Memory budget dependency: performance relies on storing 20 exemplars per class; behavior under stricter budgets is not analyzed
Added inference cost: MoE routing and cross-modal attention introduce overhead beyond standard prompt-based methods, but no latency figures are reported
Evaluated only on image classification; generalization to other CLIP use cases (retrieval, captioning, VQA) is untested
The RL training introduces hyperparameters (\(\lambda\), \(\alpha\), \(\tau\), KL weight \(\lambda_\text{KL}\)) whose sensitivity analysis appears only partially in the (truncated) paper
Food-101 gains are marginal, suggesting the approach adds less value when tasks are not fine-grained or domain-shifted

Relevance to Vision-Language Models¶

This work directly targets the deployment gap between CLIP's strong zero-shot capabilities and its fragility under sequential fine-tuning — a critical practical concern as VLMs are adapted for downstream tasks in production settings. The finding that visual prompts are systematically underexploited in continual learning is a transferable insight for any prompt-based VLM adaptation method (CoOp, MaPLe, etc.). The RLPO formulation offers a principled alternative to mean-feature prototypes that could apply broadly to any embedding-based memory system in VLM continual learning. The difficulty-aware MoE routing addresses sample heterogeneity in a modality-aware way, connecting to the broader trend of mixture-of-experts architectures in large VLMs.