Skip to content

Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting

🕒 Published (v1): 2026-04-20 10:46 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Dynamic prefix weighting for domain-class incremental VLM learning; CVPR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DPW (Dynamic Prefix Weighting) is a PEFT framework for continual learning of VLMs (CLIP) that replaces uniform prefix-tuning weights with token-level dynamic weighting. It introduces a gating module with reparameterized prefix attention and conditional activation, plus a residual weighting mechanism that routes adapter outputs only to tokens requiring additional adaptation. The method achieves state-of-the-art performance on MTIL and ODCL-CIL benchmarks while using few additional parameters.

Problem

Existing PEFT-based continual learning methods for VLMs (e.g., DIKI, MoE-Adapters) apply prefix or adapter weights uniformly across all tokens within a sample — either via softmax normalization that forces prefix weights to sum to one, or via CLS-token routing that ignores per-token task relevance. This sample-level uniformity prevents fine-grained modulation: task-relevant tokens and irrelevant tokens receive the same amount of injected task-specific information. Additionally, the conventional query-key dot product in prefix-tuning captures global attention patterns that blur inter-token distinctions, making token-wise disentanglement and cross-task knowledge transfer difficult.

Method

DPW operates within a frozen CLIP backbone and introduces three components stacked in each attention layer:

  1. Reparameterized Prefix Attention (RePA): Replaces the conventional query-key dot product \(S_{XP}^{(i)} = Q_i (K_{P_i})^\top\) with a direct affine transformation: $\(S_{XP}^{(i)} = X W_i^G + B_i^G\)$ where \(W_i^G\) and \(B_i^G\) are learnable task-specific matrices that subsume the prefix key and projection parameters. This avoids using the pretrained attention mechanism (which lacks task-specific knowledge) to compute prefix scores.

  2. Conditional Activation (CondAct): Replaces softmax on \(S_{XP}^{(i)}\) with a two-step procedure applied after sigmoid \(\sigma(\cdot)\):

  3. Conditional Normalization: normalizes prefix weights to sum to at most 1 per token (soft cap): $\(g_{ijk} = \begin{cases} \frac{\sigma(s_{ijk})}{\sum_k \sigma(s_{ijk})} & \text{if } \sum_k \sigma(s_{ijk}) \ge 1 \\ \sigma(s_{ijk}) & \text{otherwise} \end{cases}\)$

  4. Conditional Filtering: zeroes out prefixes below a per-prefix adaptive cutoff derived from a Gaussian model of CLS-token prefix scores at training time, suppressing task-irrelevant prefixes at inference.

  5. Residual Weighting Mechanism (RWM): Weights the LoRA adapter output by the residual beyond the prefix cap: $\(\Delta_{ij} = \max\!\left(0,\,\sum_{k=1}^L \sigma(s_{ijk}) - 1\right)\)$ so tokens that already receive full prefix information (\(\sum_k \sigma(s_{ijk}) < 1\)) get zero adapter contribution, while tokens with high task relevance receive proportional adapter output. The shared adapter down-projection is frozen and initialized from the top-\(k\) left singular vectors of \(W_i^V\).

Key Contributions

  • Token-level dynamic weighting of prefixes via RePA + CondAct, replacing both query-key attention and fixed softmax normalization used by prior methods.
  • CondAct's conditional normalization enforces an upper bound of 1 on total prefix weight per token, mitigating forward forgetting of zero-shot CLIP capabilities.
  • RWM that derives adapter weights as a residual of prefix weights, enabling prefix–adapter synergy without a separate router.
  • A parameter-efficient variant (Ours†) that computes prefix scores in each head's sub-dimension \(d/h'\) and uses lower-rank LoRA, reducing parameters from 30.8 M to 4.6 M.

Results

MTIL (Order I), compared to prior SOTA:

  • Ours (30.8 M params): Transfer 70.4, Avg. 79.3, Last 88.3
  • Ours† (4.6 M params): Transfer 70.0, Avg. 78.6, Last 87.6
  • Best prior method GIFT (149.6 M, extra diffusion data): Transfer 69.3, Avg. 77.5, Last 86.0
  • DIKI (1.8 M): Transfer 68.7, Avg. 76.7, Last 85.1
  • MoE-Adapter (59.6 M): Transfer 68.9, Avg. 76.9, Last 85.0

Both variants surpass all baselines including the full fine-tuning method GIFT that uses auxiliary diffusion-generated reference data.

ODCL-CIL (class-incremental, no task ID):

  • Ours: Transfer 70.4, Avg. 78.6, Last 86.6
  • Ours†: Transfer 70.0, Avg. 77.9, Last 85.8
  • Best prior (DPeCLIP): Transfer 69.1, Avg. 76.6, Last 84.6
  • MoE-Adapter degrades severely (Avg. 66.2, Last 66.9) without task IDs.

Ablation (MTIL):

  • Baseline (no DPW): Trans. 68.1, Avg. 76.6, Last 85.9
  • +RePA: Trans. 68.0, Avg. 76.8, Last 86.4
  • +RePA+CondAct: Trans. 69.5, Avg. 77.8, Last 86.8
  • +RePA+CondAct+RWM (full): Trans. 70.4, Avg. 79.3, Last 88.3

Each module contributes incrementally; RWM yields the largest gain in Average and Last.

Limitations

  • The Gaussian cutoff model for CondAct filtering is fit per task and per prefix, requiring per-task statistics maintained during training; behavior under severe distribution shift is unstated.
  • The method is evaluated only on CLIP (ViT backbone) under domain-class incremental settings; generalization to other VLM architectures (e.g., generative VLMs) is not demonstrated.
  • No evaluation of inference-time compute overhead from the per-token scoring and filtering operations.
  • Ours† reduces parameters but the principled choice of sub-dimension \(h'\) and LoRA rank is not ablated.

Relevance to Vision-Language Models

DPW directly addresses a structural weakness in adapting frozen CLIP to sequential task streams: that all PEFT methods heretofore treat prefix/adapter weighting at the sample level rather than the token level. The result is that fine-grained, token-wise routing of task-specific information — well established in vision transformers — had not been applied to the prefix-tuning paradigm for VLMs. By achieving SOTA on both MTIL and ODCL-CIL with fewer parameters than full fine-tuning baselines, DPW is immediately relevant to anyone using CLIP or prompt-tuned VLMs in continually changing deployment environments. The forward-forgetting framing (Transfer metric) also highlights a VLM-specific CL challenge — preserving zero-shot generalization — that DPW explicitly optimizes for via CondAct's weight cap.