Skip to content

AdaBoosting Text Prompts for Vision-Language Models

🕒 Published (v1): 2026-07-01 09:28 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; AdaBoosting text prompts improves zero-shot VLM classification, practical and reproducible

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Text Prompt Boosting (TPB) applies AdaBoost to few-shot text prompt adaptation of CLIP-style VLMs, treating each class-wise natural-language prompt bank as a weak learner and sequentially building an ensemble that explicitly targets misclassified "hard" examples. Unlike prior text-based methods that plateau with additional labeled shots, TPB achieves clear shot-driven gains and transfers seamlessly to larger, heterogeneous VLMs via simple re-embedding.

Problem

Existing few-shot text prompting methods (e.g., LLMbo, ProAPO) optimize a single aggregate objective such as validation accuracy, which saturates early once easy examples are correctly classified. This makes them insensitive to additional labeled shots and leaves the broader visual diversity of the target domain unexploited. Soft prompt methods (e.g., CoOp) are model-specific and cannot transfer across VLM architectures.

Method

TPB treats each prompt collection \(\mathcal{B} = \{B_1, \ldots, B_K\}\) (class-wise banks of natural-language strings) as a weak classifier whose class logit is \(s(\mathbf{x}, B_k) = \frac{1}{|B_k|}\sum_{t_k \in B_k} \cos(E_\text{img}(\mathbf{x}), E_\text{text}(t_k))\). It runs \(M\) boosting rounds under the SAMME.R multi-class AdaBoost variant:

  1. Large prompt pool construction: aggregates 80 hand-crafted CLIP templates, LLM-generated class descriptions from multiple prior works, and template–description concatenations.
  2. Greedy Prompt Composition (GPC): at each round, initializes with the single template minimizing weighted error, then greedily inserts per-class prompts from the pool (allowing duplicates for implicit upweighting) until no weighted-error reduction is possible.
  3. Boosting loop: before each GPC call, augments the few-shot set by factor \(a=4\) (random crop + flip). After GPC, updates sample weights on the original set via SAMME.R — increasing weights for misclassified examples, decreasing for correct ones.
  4. Final ensemble: aggregates all \(M\) weak classifiers via weighted real-valued voting. At inference, text embeddings are pre-computed so only one image encoder forward pass is needed.

Key Contributions

  • An AdaBoost framework operating entirely in discrete natural-language space, yielding a boosted text-prompt ensemble with strong shot scalability.
  • Greedy Prompt Composition (GPC) as the weak learner: a two-stage (template initialization + class-wise greedy insertion) algorithm that minimizes weighted classification error under current sample weights.
  • Demonstration that text-space prompt ensembles transfer across heterogeneous VLMs (different architectures, scales, training recipes) by simple re-embedding, preserving shot-driven gains where prior methods saturate.

Results

  • 11-dataset few-shot benchmarks (CLIP RN50): At 1-shot, TPB averages 63.8% vs. ProAPO 62.7% (+1.1 pp). At 16-shot, TPB reaches 70.3% vs. ProAPO 64.4% (+5.9 pp); gain from 1→16 shots is 6.5 pp for TPB vs. near-zero for ProAPO.
  • Cross-model transfer (source: ViT-B/32 → targets: ViT-L group): TPB scales from 79.66% at 1-shot to 82.07% at 16-shot (+2.4 pp). ProAPO is flat: 79.59%→80.01% (+0.4 pp). CoOp+EFT reaches 73.45% at 16-shot, well below TPB.
  • Cross-model transfer (source: ViT-B/32 → targets: ViT-H group): TPB scales from 81.73% at 1-shot to 84.24% at 16-shot (+2.5 pp). ProAPO is flat: 82.07%→82.39%.
  • ZS-CLIP baseline on ViT-L and ViT-H is 76.76% and 78.75% respectively (no few-shot), which TPB exceeds at all shot counts.

Limitations

  • CoOp with model-specific soft prompts achieves higher raw accuracy on the source model at larger shot counts (e.g., 73.2% at 16-shot vs. TPB's 70.3% on RN50); TPB trades peak source accuracy for transferability.
  • The prompt pool is pre-constructed from existing LLM-generated descriptions and fixed templates; TPB does not generate new descriptions on-the-fly, limiting its adaptability to domains absent from the pool.
  • Requires running up to \(M=50\) boosting rounds, each invoking GPC, which involves sweeping the full prompt pool per class — computational cost scales with pool size, number of classes, and boosting rounds.
  • Experiments use the text (paper is truncated); cross-model transfer results cover only CLIP-family encoders; generalization to non-CLIP VLMs (e.g., LLaVA-style models) is not evaluated.

Relevance to Vision-Language Models

TPB directly addresses the core practical challenge of adapting CLIP-style VLMs to new tasks with minimal labeled data while preserving the cross-model reusability of natural-language prompts — a property lost by all soft-prompt methods. The AdaBoost framing provides a principled mechanism to exploit few-shot diversity beyond single-metric optimization, relevant to any discrete prompt-search pipeline over LLM-generated candidates. The demonstrated transfer robustness across heterogeneous ViT-L/H VLMs (OpenCLIP, SigLIP2, DFN, EVA-02-CLIP, MetaCLIP2) is particularly significant as VLM ecosystems diversify. This work extends the line connecting ensemble learning and prompt engineering, complementing gradient-free hard-prompt methods like ProAPO.