Skip to content

Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeMul (Description-free Multi-prompt Learning) adapts CLIP for few-shot image classification by distilling knowledge directly from GPT embedding models into learnable prompts, bypassing the unreliable text-description extraction step used by prior methods. It adds a prompt-weighting mechanism to dynamically emphasize more informative prompts during training. On 11 recognition benchmarks, it outperforms both description-based and continuous prompt optimization baselines.

Problem

Existing LLM-augmented prompt learning methods for VLMs (e.g., dCLIP, WaffleCLIP) query LLMs for natural-language class descriptions and insert those into prompts. This approach inherits the high variability and low reliability of open-ended LLM text generation: responses depend on query phrasing, are often hedged ("may", "often"), and can be factually incorrect. Additionally, with multiple prompts per class, no prior method accounts for the varying semantic importance of individual prompts.

Method

DeMul has two core components:

Description-free LLM distillation: Rather than extracting text descriptions, learnable prompt vectors (in the CLIP text encoder space) are mapped into the GPT embedding space via a trainable 5-layer MLP φ, with a pseudo-inverse ψ kept frozen to preserve directional structure (a conformal/cyclic map). A distillation loss aligns φ(prompt) with GPT class-name embeddings h(c) via contrastive softmax, forcing prompts to absorb class semantics from the GPT Embedding Endpoint (specifically text-embedding-3-large, 3072-dim) without any discrete text generation.

Weighted multi-prompt learning: Each class c has M=32 learnable prompt vectors. Each prompt tᵢⱼ gets a learnable scalar weight wᵢⱼ. The classification loss is a weighted average of per-prompt CLIP match scores, with an L1 regularization term to encourage sparsity—pushing the model to rely on fewer, more informative prompts. Weights are normalized per class (sum to 1).

Training minimizes Ltotal = Lcls (weighted) + α·Ldistill + Lmapping (cyclic map pre-training on WordNet + 12 dataset class names).

Key Contributions

  • Description-free LLM distillation: replaces text-generation queries with direct alignment in GPT's embedding space, eliminating variability from open-ended generation.
  • Cyclic mapping (φ, ψ conformal map) to transfer CLIP-space prompt vectors into GPT space while preserving directional structure.
  • Learnable per-prompt importance weights with L1 regularization for sparse, effective multi-prompt classification.
  • Demonstration that stronger LLM embeddings (ada-002 < 3-small < 3-large) correlate with downstream classification accuracy.

Results

  • Average Top-1 accuracy across 11 datasets vs. GalLoP (strongest prior multi-prompt baseline):
  • 1-shot: 75.5% (DeMul) vs. 73.7% (GalLoP) — +1.8%
  • 16-shot: 85.3% (DeMul) vs. 84.5% (GalLoP) — +0.9%
  • vs. zero-shot CLIP baseline: +10.5% on 1-shot, +20.3% on 16-shot
  • Ablation (avg across 11 datasets, 16-shot): full DeMul 85.3 > w/o weighting 85.2 > w/o Ldistill 84.8 > GalLoP 84.5
  • LLM scaling (16-shot avg): ada-002 82.5 → 3-small 83.4 → 3-large 84.4
  • Outperforms dCLIP, WaffleCLIP, CoOp, MaPLe, PromptSRC, and GalLoP on most of the 11 datasets

Limitations

  • Performance still depends on few-shot training image distribution; few-shot variance is not eliminated.
  • Learnable prompt vectors are shared across all classes (not class-specific), limiting class-level specialization.
  • Relies on proprietary OpenAI Embedding API (text-embedding-3-large), introducing dependency and reproducibility concerns.
  • Improvements from each component (distillation, weighting) are individually modest; gains are incremental over a strong baseline (GalLoP).

Relevance to Vision-Language Models

This work directly advances parameter-efficient adaptation of CLIP-style VLMs, a core research thread in the VLM community. It reframes LLM-augmented prompt learning: instead of treating LLMs as text generators, it treats them as semantic embedding oracles, a cleaner interface that sidesteps the brittleness of natural-language description pipelines. The prompt-weighting mechanism addresses a gap in multi-prompt literature (e.g., ProDA, GalLoP) by making prompt importance explicit and learnable rather than uniform. The positive scaling result—stronger LLM embeddings yield better VLM downstream accuracy—suggests a principled path for improving VLM adaptation alongside LLM advances.