Skip to content

Naming to Learn: Class Incremental Learning for Vision-Language Model with Unlabeled Data

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; naming paradigm enables class incremental VLM learning from unlabeled data

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

N2L addresses class incremental learning (CIL) with vision-language models under a realistic constraint: only unlabeled images and class-name strings are available per task—no human annotations. The method combines analytic (ridge-regression) CIL with SVD-based pseudo-label refinement and a bi-level sample-reweighting scheme, formulated recursively so it provably matches joint-training performance while resisting catastrophic forgetting.

Problem

All prior CLIP-based CIL methods assume fully labeled incremental data, which is expensive in practice. Naively generating CLIP zero-shot pseudo labels and feeding them into existing CIL pipelines amplifies catastrophic forgetting because noisy hard labels destabilize the classifier update. No prior work combines pseudo-label refinement with a forgetting-free recursive incremental objective.

Method

Analytic CIL backbone. The classifier \(\hat{W}_T\) is the closed-form ridge-regression solution \(\hat{W}_T = (A_T + \lambda I)^{-1} C_T\), where \(A_t = A_{t-1} + X_t^\top M_t X_t\) and \(C_t = C_{t-1} + X_t^\top M_t Y_t\) are updated recursively, producing a solution identical to joint training and thereby eliminating forgetting.

Progressive Label Refinement (PLR). SVD is applied to the task-\(t\) feature matrix \(X_t = UDV^\top\); singular vectors whose singular values exceed threshold \(\theta=10\) are retained, projecting features to \(X_{t,k} = X_t V_k \in \mathbb{R}^{n_t \times k}\). A lightweight ridge-regression classifier \(\hat{W}_t'\) is trained on \(X_{t,k}\) with initial pseudo labels, and predictions are used to overwrite the labels (Eq. 5); this is repeated for 3 iterations. Theorem 1 formally shows that dropping directions where \(\sigma^2 \ge (2\lambda + d_j^2)(\alpha_j^*)^2\) lowers expected MSE, providing theoretical backing for the dimensionality cut.

Bi-level Weight Adjustment. - Inter-class: each sample of class \(c_i\) receives weight \(m_{\text{inter},i} = \frac{n_t}{N_{t,i} \cdot |C_t|}\), normalizing total class weight across the imbalanced pseudo-label distribution. - Intra-class: weights are sampled from \(\mathcal{N}(1, \sigma^2)\), sorted ascending, and assigned to samples in descending entropy order—high-entropy (low-confidence) samples receive lower weights.

The combined weight \(m_i = m_{\text{intra},i} \cdot m_{\text{inter},i}\) enters the weighted MSE objective, which retains a recursive closed-form solution.

Inference combines zero-shot CLIP logits with the learned classifier head via a weighted sum (following RAIL).

Key Contributions

  • New CIL setting: only unlabeled data + class-name strings per task, no annotations.
  • SVD-based progressive pseudo-label refinement with theoretical guarantee (Theorem 1) showing low-signal/low-variance directions should be dropped.
  • Bi-level weight adjustment (inter-class imbalance correction + entropy-driven intra-class confidence weighting).
  • Recursive weighted ridge-regression formulation that is mathematically equivalent to joint training on all tasks, provably mitigating forgetting.

Results

  • Evaluated on 6 benchmarks: FGVCAircraft, StanfordCars, CIFAR100, CUB200, ObjectNet, UCF under B0Inc10, B50Inc10, B0Inc20, B100Inc20 protocols; metric \(A_B\) (final-task accuracy) and \(\bar{A}\) (average across tasks).
  • N2L outperforms all baselines in every setting.
  • On distribution-shifted datasets (Aircraft, ObjectNet), margin over second-best (ENGINE): +2.75%–+8.46% in \(A_B\).
  • On Cars, CIFAR100, CUB, UCF: improvements of at least +2.82%, +0.46%, +1.77%, +3.39% over the next-best method.
  • N2L narrows the gap to the fully supervised upper bound (ground-truth labels) by ~50% on Cars, ObjectNet, and UCF.
  • N2L also outperforms CPL-enhanced variants of existing methods.
  • Backbone: LAION-400M pre-trained CLIP ViT-B/16 throughout all comparisons.

Limitations

  • CLIP backbone is frozen; the method does not update image or text encoders, limiting adaptation when the new classes are far from CLIP's pre-training distribution.
  • Pseudo-label refinement quality depends on the signal-to-noise ratio in the initial CLIP zero-shot predictions; very low zero-shot accuracy (e.g., Aircraft at ~26%) leaves room that refinement only partially recovers.
  • Threshold \(\theta = 10\) for SVD truncation is fixed empirically; sensitivity analysis is provided but automatic selection is not derived.
  • Class-name strings must be available upfront per task; structured or compositional class naming variations are not studied.
  • Evaluation is limited to image classification; downstream tasks such as detection or segmentation are not explored.

Relevance to Vision-Language Models

This paper advances the use of CLIP as a frozen feature extractor and zero-shot labeler within a continual-learning loop, a setting directly relevant to deploying VLMs in open-world scenarios where annotation is infeasible. The work shows that CLIP's text-image alignment is powerful enough to bootstrap an incremental classifier without any labels, provided label noise is aggressively managed—relevant to anyone studying semi-supervised or unsupervised adaptation of VLMs. The analytic recursive framework is a computationally lightweight alternative to prompt-tuning or adapter-based CIL, making it attractive for resource-constrained VLM deployment. Results also reveal that unimodal ViT-based CIL methods degrade badly when cross-modal alignment is ignored, underscoring that CIL research must treat VLMs as inherently bimodal systems.