Skip to content

VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VaMP is a variational prompt learning framework for CLIP-based VLMs that replaces fixed, shared prompt tokens with instance-conditioned latent variables sampled from a learned posterior distribution. It introduces layer-wise Gaussian posteriors over text-side prompts, conditioned on image features, and regularizes them with a class-aware prior built from in-class prototype embeddings. On NeurIPS 2025 benchmarks, it achieves state-of-the-art across base-to-new generalization, domain generalization, and cross-dataset transfer.

Problem

Existing multi-modal prompt learning methods (MaPLe, MMRL, CoCoOp) learn fixed, task-level prompt tokens shared uniformly across all inputs. This deterministic design cannot capture instance-level variation or model uncertainty, limiting generalization under distribution shifts and in few-shot regimes. Prior uncertainty-aware methods (Bayesian Prompt Learning, Any-Shift Prompting) are restricted to text-only, globally-shared latent variables and use uninformative isotropic Gaussian priors, missing fine-grained cross-modal and inter-class semantic structure.

Method

VaMP has three components operating on a frozen ViT-B/16 CLIP backbone:

  1. Sample-specific multi-modal prompt generation: For each input image x, a frozen CLIP image encoder extracts f_x. Layer-specific lightweight MLPs {Φ_i} map f_x to dynamic text-side prompt tokens z_i ∈ R^{M×d} inserted at each of H transformer layers in the text encoder. Vision-side prompts are static and shared (not stochastic).

  2. Variational prompt adaptation (VMPA): Text prompts are treated as latent variables. Layer-specific inference networks {ϕ_i} predict per-layer Gaussian parameters [μ_i, σ_i] = ϕ_i(f_x), yielding posterior q_ϕ(z_i | x) = N(μ_i, diag(σ_i²)). Prompts are sampled via the reparameterization trick z_i = μ_i + σ_i ⊙ ε_i. Training maximizes a layer-summed ELBO.

  3. Class-aware prior: Instead of a standard N(0,I) prior, class prototypes o_y (mean of frozen image embeddings over training instances of class y) are fed through per-layer prior networks {ψ_i} to produce class-conditioned Gaussians p_ψ(z_i | o_y). The KL term in the ELBO is computed against this structured prior. At test time (no label available), the prior reverts to N(0,I). Inference averages predictions over S=10 Monte Carlo samples.

Key Contributions

  • Token-wise variational modeling across multiple intermediate transformer layers (not a single global latent), enabling hierarchical, fine-grained uncertainty capture.
  • Multi-modal posterior inference: text-side prompt distributions conditioned on visual features, improving cross-modal alignment over text-only variational methods.
  • Class-aware prior from in-class prototype embeddings, replacing the uninformative isotropic Gaussian and providing discriminative regularization of the latent prompt space.
  • End-to-end training via reparameterized sampling; inference-time MC averaging over S=10 samples for robustness.

Results

  • Base-to-new generalization (11 datasets, 16-shot): VaMP achieves average novel accuracy 78.67% (H=82.37%), outperforming MMRL (best prior) by +1.51% on novel classes. DTD novel: 75.50% vs MMRL 72.83% (+2.67%).
  • Domain generalization (ImageNet→-V2/-S/-A/-R): Average target accuracy 61.73%, +1.20% over MMRL (60.53%). ImageNet-Sketch: 49.69% vs MMRL 49.17%.
  • Cross-dataset generalization (ImageNet→10 datasets): Average 67.74% vs MMRL 67.25% (+0.49%). EuroSAT: 53.82% vs MMRL 53.10%.
  • Ablations (averaged across 11 datasets on H score): sample-specific prompts add ~+1.3% H over fixed prompts; variational modeling adds ~+0.5–1.0% H over deterministic; class-aware prior adds ~+0.5% H over standard Gaussian prior. All three components contribute additively.

Limitations

  • Vision-side prompts remain static and shared; only text-side prompts are modeled probabilistically, leaving visual uncertainty unaddressed.
  • Class-aware prior construction requires averaging over training-set image embeddings offline; unavailable at test time for unseen classes, forcing reversion to uninformative prior.
  • Inference requires S=10 forward passes for MC averaging, increasing test-time compute by ~10× over deterministic baselines.
  • Evaluated only on CLIP ViT-B/16; generalization to other VLM architectures or larger CLIP variants is not demonstrated.
  • Improvements over the strongest baseline (MMRL) are modest (0.49–1.51%), raising questions about marginal utility on in-distribution tasks.

Relevance to Vision-Language Models

VaMP directly advances CLIP prompt adaptation—the dominant paradigm for parameter-efficient VLM fine-tuning—by bringing variational inference into the multi-modal prompt space. The key insight that treating prompt tokens as per-layer, image-conditioned latent variables (rather than global point estimates) improves generalization connects to the broader challenge of making VLMs robust to domain shift without full fine-tuning. The class-aware prior mechanism is a concrete technique for injecting semantic structure into learned prompt distributions, relevant to any work on controllable or class-discriminative VLM adaptation. For researchers tracking VLMs, this paper establishes a new best practice for uncertainty-aware prompt tuning and provides strong ablation evidence that instance-level stochasticity and structured priors each contribute meaningfully.