Enhanced Continual Learning of Vision-Language Models with Model Fusion¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Model fusion enhances continual VLM learning; mitigates catastrophic forgetting

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ConDU (Continual Decoupling-Unifying) introduces model fusion into continual learning for VLMs (CLIP), maintaining a single unified model that can be decoupled into per-task experts via lightweight task triggers. It eliminates catastrophic forgetting without replay buffers, reference datasets, or trade-off hyperparameters, while simultaneously enhancing zero-shot generalization beyond the original pretrained VLM.

Problem¶

Sequential fine-tuning of VLMs on multiple downstream tasks causes catastrophic forgetting and degrades zero-shot capabilities. Existing VLM continual learning methods require: (a) external reference datasets for distillation and sensitive hyperparameter tuning (ZSCL, Dual-RAIL), or (b) restriction to parameter-efficient fine-tuning only (adapter/LoRA-based methods). No prior work supports both full fine-tuning and PEFT while avoiding all three drawbacks simultaneously.

Method¶

ConDU operates over delta models $\delta^t = \theta^t - \theta^0$ (fine-tuned minus pretrained weights). At each session $t$:

Tune Individually: Fine-tune the pretrained VLM $\theta^0$ on task $t$ to obtain task expert $\theta^t$ and its delta $\delta^t$.
Unify Models: Merge all delta models $\{\delta^1, \ldots, \delta^t\}$ into a unified delta $\delta^{1:t}$ via an element-wise election: $$\delta^{1:t}_j = \begin{cases} \max_i(\delta^i_j) & \text{if } \sum_i \delta^i_j > 0 \\ \min_i(\delta^i_j) & \text{if } \sum_i \delta^i_j < 0 \end{cases}$$ Each task $i$ produces a task trigger $(M^i, \lambda^i)$: a binary sign-agreement mask $M^i_j = \mathbf{1}[\delta^i_j \cdot \delta^{1:t}_j > 0]$ and a magnitude rescaler $\lambda^i = \frac{\text{sum}(|\delta^i|)}{\text{sum}(|M^i \odot \delta^{1:t}|)}$.
Decouple Unified Model: Reconstruct any past task expert as $\tilde{\delta}^i = \lambda^i \cdot M^i \odot \delta^{1:t}$, keeping only one unified model and $t$ small trigger pairs in memory.

At inference, for zero-shot or task-agnostic settings, a Semantic-based Aggregating Mechanism computes cosine similarity between the test image feature (from $\theta^0$) and stored per-category prototypes (mean of image + text features from $\theta^0$), selects the top-$K$ task experts, and aggregates their logits with binary weights. Decoupling and unifying are training-free and take ~1% of the time of fine-tuning.

Key Contributions¶

First application of model fusion to VLM continual learning; works under both full fine-tuning and PEFT (LoRA).
Training-free decoupling/unifying via sign-based task triggers eliminates need for reference datasets, replay buffers, and trade-off hyperparameter tuning.
Semantic-based aggregation with prototype matching for task-agnostic and zero-shot inference.
Demonstrated improvement over zero-shot baseline: ConDU enhances original VLM zero-shot Transfer by +5.5% on standard MTIL.

Results¶

MTIL benchmark (11-domain, task-incremental, vs. ZSCL/Dual-RAIL/DPeCLIP/MulKI baselines): - Transfer (zero-shot on future tasks): ConDU (FT) 70.8% vs. best baseline 70.1% (+0.7%); vs. pretrained CLIP 65.3% (+5.5%) - Average (across all sessions): ConDU (FT) 78.8% vs. best baseline 77.8% (+1.0%); vs. pretrained 65.3% (+13.5%) - Last (end of training): ConDU (FT) 87.1% vs. best baseline 86.9% (+0.2%)

Task-agnostic MTIL (no task ID at inference, vs. ZSCL/DPeCLIP/CoLeCLIP et al.): - Average: ConDU (FT) 78.1% vs. best baseline 76.1% (+2.0%); vs. pretrained 57.8% (+20.3%) - Last: ConDU (FT) 86.4% vs. best baseline 84.6% (+1.8%); vs. pretrained 57.8% (+28.6%)

Few-shot MTIL (5 samples/class): - Transfer: ConDU (LoRA) 70.3% vs. best baseline 68.9% (+1.4%) - Average: ConDU (LoRA) 72.7% vs. best baseline 71.4% (+1.3%) - Last: ConDU (LoRA) 77.4% vs. best baseline 71.4% (+6.0% over MoE)

Limitations¶

Stores $t$ task triggers (mask + scalar per task) that grow linearly with the number of tasks; at large $t$ this may become non-trivial.
The unifying election rule approximates individual delta models ($\delta^i \approx \tilde{\delta}^i$); approximation error accumulates as $t$ grows, a potential scalability concern not studied beyond 11 tasks.
Prototype storage scales with number of categories across all tasks.
$K$ (number of task experts activated at inference) is the sole remaining hyperparameter; though ablations show low sensitivity, its optimal value is not derived analytically.
Evaluation confined to CLIP-based VLMs on MTIL; generalization to other VLM architectures (e.g., LLaVA, InstructBLIP) is not demonstrated.

Relevance to Vision-Language Models¶

ConDU directly addresses a core deployment challenge for CLIP-family VLMs: how to specialize them on private or domain-specific tasks sequentially without sacrificing the zero-shot generalization that makes them broadly useful. Unlike standard continual learning adapted from unimodal settings, ConDU is designed around the delta-model arithmetic that arises naturally from pretrained-to-finetuned weight differences, a property specific to large pretrained VLMs. The result — a single stored model that recovers arbitrarily many task experts on demand — is a practical memory-efficient solution for multi-domain VLM deployment, and its compatibility with LoRA makes it immediately relevant to the PEFT-for-VLMs line of work (e.g., CLIP-Adapter, CoCoOp successors).