Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Null space adaptation enables memory-free continual VLM learning; preserves zero-shot

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NuSA-CL is a memory-free continual learning framework for CLIP-like VLMs that constrains weight updates to the intrinsic null space of the model's current parameters, identified via SVD before each new task. By merging low-rank null-space updates directly into the backbone, it preserves zero-shot generalization without replay buffers, gradient memories, or parameter growth. It achieves state-of-the-art performance among storage-free methods on MTIL and CIFAR-100 class-incremental benchmarks.

Problem

Pre-trained VLMs like CLIP have static zero-shot capabilities that degrade under continual learning due to catastrophic forgetting. Existing continual learning approaches either require storage that grows with task count (replay buffers, gradient projection memories, expanding adapter libraries) or cause parameter bloat (task-specific modules), making them impractical for lifelong deployment in resource-constrained settings. Even PEFT methods like InfLoRA still require external gradient projection memory to enforce orthogonality.

Method

NuSA-CL operates in a three-stage cycle per task:

1. Null Space Identification: Given current weights \(W_{t-1} \in \mathbb{R}^{m \times n}\), compute SVD \(W = U\Sigma V^\top\). Identify the principal subspace dimension \(k\) as the smallest integer satisfying \(\sum_{i=1}^{k} \sigma_i^2 \geq \rho \cdot \|W\|_F^2\). The remaining \(d - k\) directions span the intrinsic null space, with basis \((U_n, V_n)\).

2. Constrained Adaptation: Instead of standard LoRA, define the update as \(\Delta W = U_n M V_n^\top\) where \(U_n, V_n\) are frozen and only the small intermediate matrix \(M \in \mathbb{R}^{r \times r}\) (initialized to zero) is trained. The effective rank is \(r = \min(d - k, r_{\max})\) with \(r_{\max} = 128\). This guarantees \(\Delta W\) is mathematically orthogonal to the principal subspace of \(W\).

3. Weight Merging: Merge directly into the backbone: \(W_t \leftarrow W_{t-1} + \Delta W_t\), maintaining a fixed parameter budget. The cycle repeats from the updated \(W_t\) for the next task.

The theoretical guarantee (Theorem 2) bounds cumulative interference: \(\sum_{t=1}^{T} |\langle W_{t-1}, \Delta W_t \rangle_F| \leq \sum_{t=1}^{T} \sigma_{t,\max}^{\text{null}} \cdot \|M_t\|_F\), where \(\sigma_{t,\max}^{\text{null}}\) is the largest singular value in the null subspace — which is small by construction.

Key Contributions

• Memory-free continual learning framework with zero storage overhead, zero auxiliary model load, and zero parameter growth.
• Null space-constrained LoRA update (\(\Delta W = U_n M V_n^\top\)) that is persistently confined to low-energy spectral directions throughout training, not merely initialized there.
• Theoretical bound on cumulative parameter-space interference across \(T\) sequential tasks.
• Empirical demonstration that the null space is not exhausted even after 50 tasks: the most saturated layer retains ~313 available null directions vs. \(r_{\max} = 128\).

Results

• MTIL (full-shot), storage-free setting: NuSA-CL achieves Transfer/Avg./Last of 68.6/75.1/82.8%, vs. LoRA at 63.9/70.1/79.9% and MiLoRA at 62.8/68.7/77.4%.
• MTIL vs. storage-based SOTA: NuSA-CL (1.21 GPU-hours, 1.5M params, 6.6 GB peak GPU) is highly competitive with MoE-Adapters (85.0% Last, 3.42 GPU-hours, 59.8M params) — 40× fewer parameters, ~3× faster.
• MTIL (5-shot), storage-free PEFT setting: NuSA-CL achieves best Transfer, Avg., and Last across all 11 datasets, outperforming InfLoRA (storage-based) by up to ~2% on average metrics.
• CIFAR-100 class-incremental, 50 steps: NuSA-CL Last = 71.85%, vs. ZSCL at 67.36% (+4.49%), while using a fraction of ZSCL's compute.
• Ablation on subspace selection: Null (tail) subspace consistently yields lower forgetting than top or random subspaces across all update ranks.

Limitations

• Theoretical forgetting bounds are in parameter space, not function space; tighter output-level guarantees are left as future work.
• Null space is approximated (low-energy, not strictly zero singular values), so some interference with principal components remains.
• Evaluated on CLIP ViT-B/16 only; generalization to larger VLM backbones (ViT-L, ViT-G) or multimodal LLMs (LLaVA, VLA models) is asserted but not empirically demonstrated.
• The fixed \(r_{\max} = 128\) cap is a hyperparameter whose optimal value may shift across architectures or task lengths.

Relevance to Vision-Language Models

NuSA-CL directly targets CLIP's visual and text encoders — the perceptual backbone of downstream MLLMs and VLA models — enabling them to absorb new visual distributions without corrupting pre-trained zero-shot alignment. This is practically important for VLM deployment in open-world settings where new classes or domains emerge after model release, a scenario where standard fine-tuning silently degrades cross-modal representations. The method's fixed parameter budget and no-replay design make it uniquely suited for on-device or edge-deployed VLMs, complementing higher-level continual instruction tuning of decoder heads. For VLM researchers, the spectral analysis showing progressive null-space utilization offers a new diagnostic lens for understanding how CLIP representations change under sequential adaptation.