KeepLoRA: Continual Learning with Residual Gradient Adaptation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

KeepLoRA: residual gradient adaptation balances VLM plasticity and stability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

KeepLoRA is a continual learning method for pre-trained VLMs that restricts LoRA parameter updates to the residual subspace of the weight matrices—orthogonal to both the pre-trained model's principal subspace and dominant feature directions of all previously learned tasks. This unified subspace constraint prevents interference with general pre-trained knowledge (forward stability) and previously learned task knowledge (backward stability) while a gradient-informed initialization preserves plasticity. It achieves state-of-the-art results on the MTIL benchmark without any inference overhead.

Problem¶

Continual fine-tuning of VLMs on a sequence of tasks suffers from three competing pressures: acquiring new task knowledge (plasticity), retaining previously learned tasks (backward stability), and preserving the general zero-shot transferability encoded in pre-training (forward stability). Existing LoRA-based CL methods (O-LoRA, InfLoRA, SD-LoRA) address backward stability but ignore the preservation of pre-trained knowledge, and use suboptimal update directions that hurt plasticity. Reference-data regularization methods depend on external datasets and increase compute cost; architecture extension methods add inference overhead.

Method¶

The method rests on an SVD analysis showing that high-energy singular components of pre-trained weight matrices encode general knowledge, while low-energy (residual) components encode domain-specific knowledge. KeepLoRA therefore confines all LoRA updates to the residual subspace.

Unified principal subspace construction. For each weight matrix $W = USV^\top$, the principal subspace $W_p = U_{:,1:p}$ is retained under energy threshold $\epsilon_w$: $\|W_p\|_F^2 \geq \epsilon_w \|W\|_F^2$. After training task $t$, dominant feature directions $V_{t,1:m}$ of the projected features $\hat{X}_t = X_t - W_p W_p^\top X_t - M_{t-1}M_{t-1}^\top X_t$ are appended to the accumulated matrix $M_t$. The unified subspace is $M_t' = [W_p, M_t]$.

Gradient-informed LoRA initialization. At the first step of task $t$, the gradient $G_t = \nabla_W \mathcal{L}(W; D_t)$ is projected onto the residual subspace: $$\hat{G}_t = G_t - W_p W_p^\top G_t - M_{t-1} M_{t-1}^\top G_t$$ SVD of $\hat{G}_t = USV^\top$ gives LoRA matrices $A = U_{:,1:r}$, $B = S_{1:r} V_{:,1:r}^\top$. Matrix $A$ is then frozen; only $B$ is trained. The pre-trained weight is offset as $W' = W - \frac{\alpha}{r}AB$ so initial behavior is unchanged. The theoretical justification (Propositions 3.1–3.2) shows frozen-$A$ updates are equivalent to gradient projection onto $\text{span}(A)$, and that the initialization solves $\min_{A^\top A=I, W_p^\top A = M_{t-1}^\top A = 0} \|G_t - AA^\top G_t\|_F^2$—the optimal residual-subspace approximation to the full-fine-tuning gradient.

Key Contributions¶

Empirical finding that principal singular components encode general VLM knowledge while residual components encode domain-specific knowledge, validated by ablation over backbone reconstruction.
KeepLoRA algorithm: gradient-projected, residual-subspace-constrained LoRA with frozen $A$ and gradient-informed initialization; no inference overhead (adapters merge into weights).
Theoretical proof that frozen-$A$ LoRA is equivalent to gradient projection, and that the initialization is the solution to a constrained optimization maximizing gradient capture in the residual subspace.
Validated on both dual-encoder (CLIP) and encoder-decoder (LLaVA) architectures on the MTIL benchmark.

Results¶

All scores on the MTIL benchmark (10 classification tasks, task-incremental setting):

Average accuracy: KeepLoRA 77.5% vs. InfLoRA 76.3%, SD-LoRA 72.5%, O-LoRA 73.3%, ZSCL 75.4%
Last accuracy (final task): KeepLoRA 86.1% vs. InfLoRA 85.2%, SD-LoRA 79.2%, ZSCL 83.6%
Transfer (zero-shot degradation): KeepLoRA 55.6 vs. InfLoRA 53.6, O-LoRA 39.8 (lower transfer loss is better)
KeepLoRA+ (with architecture extension): Average 79.0%, Last 88.2%—best in the architecture-extension group, outperforming IAP (76.8% / 85.7%) and MoE-Adapters (76.7% / 85.0%)
KeepLoRA achieves the best Trade-off: highest Transfer score while also leading Average and Last among reference-data methods

Limitations¶

The unified principal subspace $M_t'$ grows with each task (up to $d_{in}$ orthogonal directions); memory cost increases over long task sequences.
Computing the first-step gradient $G_t$ requires a forward+backward pass before training begins, adding modest initialization overhead.
Energy thresholds $\epsilon_w$ and $\epsilon_f$ are hyperparameters that must be tuned; sensitivity analysis is performed but optimal values may vary across architectures.
Evaluation is restricted to the MTIL classification benchmark; generative tasks (e.g., VQA, captioning with LLaVA) are mentioned but not fully benchmarked in the main results.
The method assumes task identity is known at inference for task-incremental evaluation; performance under class-incremental (task-agnostic) settings is not reported.

Relevance to Vision-Language Models¶

KeepLoRA directly targets a core challenge for deploying VLMs in real-world, multi-domain pipelines: how to specialize a CLIP or LLaVA model to successive tasks without destroying its general zero-shot capabilities. The SVD-based subspace analysis provides a mechanistic explanation for why naive LoRA fine-tuning degrades transfer, which is broadly relevant to any practitioner fine-tuning VLMs. The approach is architecture-agnostic (demonstrated on both dual-encoder CLIP and encoder-decoder LLaVA) and incurs zero inference overhead by merging adapters—a practical advantage over prompt-pool or MoE adapter approaches common in the VLM continual learning literature. For researchers tracking VLMs, KeepLoRA represents the current state-of-the-art gradient-projection baseline for multi-domain task-incremental learning.