Skip to content

PRiSM: Prototype Regularization for Few-Shot VLMs

🕒 Published (v1): 2026-07-20 11:08 UTC · Source: Arxiv · link

Why this paper was selected

Dolz + Ben Ayed; few-shot VLM adaptation under realistic class-imbalance; buildable recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PRiSM is a plug-and-play prototype regularization module for training-free few-shot adaptation of VLMs that corrects class prototype bias induced by class imbalance. It optimizes a multi-term loss combining support-feature alignment, fidelity to baseline prototypes, and inter-class pairwise separation, solved via a block Majorize-Minimize algorithm with an analytically derived step size. On a newly proposed Dirichlet-sampled benchmark with realistic imbalance, PRiSM consistently lifts all tested baselines, with gains up to +35 mean accuracy points.

Problem

Standard few-shot VLM benchmarks assume class-balanced support and query sets, which masks a fundamental failure mode: under realistic imbalance, majority classes dominate prototype estimation and additional labeled samples amplify this bias rather than reducing it. Existing training-free methods (TIP-Adapter, APE, GDA, ProKeR) degrade severely as shot count increases when class frequencies are skewed, yet this degradation is invisible under the canonical balanced protocol.

Method

PRiSM learns rectified class prototypes \(w = [w_1, \ldots, w_C] \in \mathbb{R}^{dC}\) by minimizing:

\[\mathcal{L}(w, \phi) = \beta \sum_c \|w_c - \mu_c(\phi)\|_2^2 + \gamma \sum_c \|w_c - a_c\|_2^2 - \frac{\lambda}{2(C-1)} \sum_{c \neq c'} \|w_c - w_{c'}\|_2^2\]

where \(\mu_c(\phi)\) is the mean of support-set visual embeddings for class \(c\), \(a_c\) is the prototype produced by any baseline method, and the concave third term drives inter-class separation. Two blocks of variables are optimized alternately via block Majorize-Minimize (MM): (1) prototype updates \(w\) with implicit step size \(1/\rho\), where \(\rho = 2(|\beta+\gamma-\lambda|+\lambda)\) is derived analytically via the Gershgorin circle theorem applied to the Kronecker-structured Hessian \(\Phi = R \otimes I_d\); (2) lightweight LoRA fine-tuning of the last three layers of the CLIP vision encoder (rank 8) to adapt visual means \(\mu_c(\phi)\). The optimizer requires no validation-set step-size search. All experiments use a single fixed hyperparameter configuration (\(\beta=0.01, \gamma=1, \lambda=0.05\)) across all datasets.

Key Contributions

  • A new realistic few-shot benchmark using Dirichlet sampling (\(\delta \in [0.1, 0.3]\) for severe imbalance, \(\delta \approx 0.9\) for near-balanced) with two class-coverage regimes (10–30% and 70–90%), revealing that balanced benchmarks conceal severe prototype bias.
  • Empirical finding that increasing shots under imbalanced conditions degrades several state-of-the-art training-free methods.
  • PRiSM: a validation-free, plug-and-play prototype rectification module applicable on top of any few-shot VLM baseline.
  • A block MM optimizer with analytically derived blockwise Lipschitz constant via the Gershgorin circle theorem, avoiding step-size hyperparameter search.

Results

  • 16-shot, severe imbalance, high class coverage (mean over 10 datasets):
  • Tip-Adapter: 33.25 → 68.30 (+35.05)
  • APE: 54.15 → 68.74 (+14.59)
  • GDA: 57.64 → 64.20 (+6.56)
  • ProKeR: 71.40 → 74.43 (+3.03)
  • 4-shot, severe imbalance: Tip-Adapter 49.64 → 72.86 (+23.22); APE 63.58 → 71.23 (+7.65)
  • After rectification, Tip-Adapter (simple cache method) reaches parity with ProKeR (CVPR'25 kernel-based method): e.g., 72.19 vs. 72.31 at 2-shot; 72.86 vs. 71.56 at 4-shot
  • Near-balanced setting: PRiSM also improves baselines, though gains are smaller (+0.4 to +6.6 at 16-shot per Fig. 1)
  • Single hyperparameter configuration used across all 10 evaluation datasets with no per-task tuning

Limitations

  • Improvements on the strongest baseline (ProKeR) are modest under 2-shot near-balanced conditions, and a few individual dataset/shot combinations show small negative deltas (e.g., ProKeR on Food-101 at 4-shot: −3.68).
  • The vision encoder LoRA fine-tuning introduces a gradient-based optimization step, breaking the fully training-free nature of the underlying baselines.
  • Evaluation is restricted to CLIP ViT-B/16; generalization to larger VLM architectures (e.g., LLaVA, Flamingo) is not demonstrated.
  • The Dirichlet-sampled benchmark covers only image classification; extension to other VLM tasks (retrieval, captioning, VQA) is not addressed.
  • \(\lambda > 0\) makes the prototype-update subproblem non-convex; convergence guarantees are to local minima only.

Relevance to Vision-Language Models

PRiSM directly addresses a gap in how few-shot adaptation of VLMs like CLIP is evaluated and performed, revealing that the standard balanced benchmark flatters existing methods and hides their brittleness to real-world label skew. The work is relevant to the growing body of training-free CLIP adaptation (TIP-Adapter, APE, GDA, ProKeR) by providing a model-agnostic rectification layer that can be stacked on any such method. The finding that prototype quality—not the adaptation mechanism—is the primary performance bottleneck under imbalance has implications for how the community should design and report few-shot VLM experiments. The combination of LoRA-based encoder adaptation with analytically derived optimization (no validation set) is a practically significant contribution for deployment scenarios where labeled validation data is unavailable.