Pi-CCA: Prompt-Invariant CCA Certificates for Replay-Free Continual Multimodal Learning¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; prompt-invariant CCA certificates enable replay-free continual VL learning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Pi-CCA addresses catastrophic forgetting of cross-modal alignment in continual vision-language model adaptation by replacing proxy-signal regularization with direct preservation of CCA geometry—top-$k$ canonical correlations and subspaces—stored as a compact, constant-memory certificate. A prompt-invariance loss, computed by averaging canonical subspace projectors over randomized prompt perturbations, further reduces brittleness to phrasing shifts. The method is replay-free, generator-free, and LoRA-compatible, achieving state-of-the-art results on four VL-CL benchmarks.

Problem¶

Standard fine-tuning of VLMs on non-stationary streams causes catastrophic forgetting of zero-shot cross-modal generalization and sensitivity to prompt/style variation. Prior replay-free VL-CL methods regularize proxy quantities (logits, similarities, parameters, routing weights) rather than the underlying alignment geometry, leading to slow drift in the whitened image–text cross-covariance subspace, dependency on reference corpora or teacher ensembles, and residual prompt brittleness.

Method¶

Pi-CCA constructs a compact CCA certificate $\{\rho^\star_{1:k},\, S^\star_v,\, \bar{S}^\star_t\}$ from a pre-continual CLIP checkpoint: - Spectral invariants: top-$k$ canonical correlations $\rho^\star_{1:k}$ of the whitened cross-covariance $\widehat{M} = \hat{\Sigma}_{vv}^{-1/2}\hat{\Sigma}_{vt}\hat{\Sigma}_{tt}^{-1/2}$. - Directional invariants: sketched canonical subspace bases $S^\star_v = R_v^\top U^\star_k \in \mathbb{R}^{h\times k}$ and $\bar{S}^\star_t$ (prompt-averaged), with random orthonormal sketches $R_v \in \mathbb{R}^{d_v\times h}$, $h\ll d_v, d_t$, giving $O(h k)$ storage. - Prompt-invariant certificate: text subspace projectors $Q^\star_t(\delta_m) = R_t^\top P^\star_t(\delta_m)R_t$ are averaged over $M$ prompt perturbations and their top-$k$ eigenvectors taken as $\bar{S}^\star_t$, eliminating sign/rotation ambiguity without Procrustes alignment.

During training, only LoRA parameters $\phi_v,\phi_t$ are updated via four losses: $$\mathcal{L} = \mathcal{L}_\text{task} + \lambda_1\mathcal{L}_\text{spec} + \lambda_2\mathcal{L}_\text{sub} + \lambda_3\mathcal{L}_\text{pi}$$ - $\mathcal{L}_\text{spec}$: sorted $\ell_2$ distance between current and certificate canonical correlations plus Ky-Fan-$k$ sum alignment. - $\mathcal{L}_\text{sub}$: Frobenius distance between sketched Gram projectors $\hat{Q}_v, \hat{Q}_t$ and certificate counterparts $Q^\star_v, \bar{Q}^\star_t$. - $\mathcal{L}_\text{pi}$: aligns mean perturbed projector to $\bar{Q}^\star_t$ and contracts within-perturbation dispersion.

Mini-batch covariances are stabilized via EMA: $\Sigma^{(t)}_{\cdot} \leftarrow (1-\beta)\Sigma^{(t-1)}_{\cdot} + \beta\hat{\Sigma}_{\cdot}$. The certificate itself is slowly EMA-refreshed with rate $\alpha \ll 1$, using Newton–Schulz or eigendecomposition-based stable whitening and block power iteration for differentiable SVD.

Key Contributions¶

Recasting VL-CL forgetting as alignment-geometry drift rather than proxy-signal mismatch, with a principled fix that directly constrains canonical correlations and subspaces.
A replay-free, constant-memory CCA certificate ($O(hk)$ storage independent of $d_v, d_t$) that works without reference corpora, generators, or task metadata.
A prompt-invariance mechanism (projector averaging over perturbations + $\mathcal{L}_\text{pi}$) that reduces sensitivity to synonym/template variation under both ID and OOD prompts.
Empirical evidence linking geometry preservation (angle drift $D_\text{ang}$, spectral drift $D_\rho$) to downstream retention, with Pearson/Spearman $\approx 1.00$ across ablated configurations.

Results¶

MTIL (11-domain classification): - Pi-CCA: Avg 76.8, Last 75.5, Transfer 73.2 — best replay-free; next best C-CLIP: 75.2/73.8/70.9.

X-TAIL (task-agnostic cross-domain): - Pi-CCA: Avg 68.1, Last 66.9, Transfer 64.7 — best replay-free; next best RAIL: 67.4/66.2/64.2.

VLCL retrieval (8-task sequential image–text retrieval): - Pi-CCA: I2T R@1 48.6±1.0, T2I R@1 37.4±0.8 — surpasses all replay-free methods and even synthetic-replay GIFT (47.3/36.5).

ConStruct-VL (structured VL concepts, 7 tasks): - Pi-CCA: FA 75.2±1.3, AF 2.7±0.2 — highest FA and lowest forgetting; next best GIFT†: 73.9/3.3.

Ablations (MTIL Avg drop when component removed): - No spectral term ($\lambda_1=0$): −2.5 pp; no subspace term ($\lambda_2=0$): −2.2 pp; no covariance EMA ($\beta=0$): −2.7 pp; no prompt invariance: −1.5 pp.

Task-order robustness: Narrow IQR over 20 shuffled MTIL sequences; between-order span modest.

Limitations¶

Requires computing mini-batch SVD and whitening at each step, which adds wall-clock overhead compared to simpler distillation losses (though shown to be manageable on A100-80GB at ~270–320 ms/step).
Prompt-invariance requires a diverse anchor prompt set at initialization; the quality of $\bar{S}^\star_t$ depends on prompt diversity.
The Pareto-optimal certificate size (k=64, h=256) was tuned on MTIL; transfer to substantially different modalities or architectures may require re-calibration.
Evaluation is limited to CLIP-family VLMs with LoRA adaptation; applicability to larger generative VLMs (e.g., LLaVA, Flamingo) is not demonstrated.
The prompt stress test suggests a practical operating range of $s \leq 0.6$; very aggressive OOD prompt shifts still degrade performance.

Relevance to Vision-Language Models¶

Pi-CCA directly targets a core failure mode of continual VLM deployment: erosion of CLIP-style zero-shot cross-modal alignment under sequential domain shifts without data replay. By grounding the preservation objective in canonical correlation geometry rather than contrastive logits, it offers a theoretically motivated alternative to distillation-based methods (ZSCL, C-CLIP, Mod-X) that dominate this literature. The prompt-invariance component addresses a practical gap—brittleness to template variation—that is orthogonal to forgetting and thus broadly relevant to any VLM deployment scenario requiring robust zero-shot generalization. The constant-memory, generator-free design lowers the barrier for real-world continual VLM adaptation under privacy or resource constraints.