Skip to content

Divergence-enhanced Knowledge-guided Context Optimization for Visual-Language Prompt Tuning

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeKg (Divergence-enhanced Knowledge-guided Prompt Tuning) addresses the pre-training bias introduced by consistency-based knowledge-guided prompt tuning for CLIP. It adds an HSIC (Hilbert-Schmidt Independence Criterion) regularization term that penalizes statistical dependence between learnable and crafted prompts, encouraging task-specific divergence while preserving generalization. DeKg is parameter-free, plug-and-play, and consistently improves base-to-new, cross-dataset, and few-shot performance over strong baselines.

Problem

Knowledge-guided context optimization (KGCO) methods like KgCoOp prevent catastrophic forgetting in CLIP prompt tuning by minimizing the distance between learnable and hand-crafted prompts. This consistency constraint introduces a bias toward the pre-trained distribution, which degrades performance on base (seen) classes when the downstream data distribution differs from pre-training data—creating a tension between generalization to new classes and discriminability on base classes.

Method

DeKg augments existing KGCO methods with an independence regularization term \(\mathcal{L}_{kd} = \text{HSIC}(W, W_{\text{clip}}) = (N_c-1)^{-2}\,\text{tr}(KHK_{\text{clip}}H)\), where \(K\) and \(K_{\text{clip}}\) are Gram matrices of learnable and crafted textual embeddings, and \(H\) is the centering matrix. Using an inner-product kernel (\(K_{i,j} = w_i^\top w_j\)), minimizing \(\mathcal{L}_{kd}\) reduces statistical dependence between the two prompt sets. Crucially, because \(K_{\text{clip}}\) is fixed (frozen CLIP embeddings), \(\mathcal{L}_{kd}\) operates by enhancing inter-class divergence among learnable prompts guided by the inter-class structure of crafted prompts. The full objective is \(\mathcal{L} = \mathcal{L}_{ce} + \lambda\mathcal{L}_{kg} + \mu\mathcal{L}_{kd}\) with \(\lambda/\mu = 3/1\). DeKg is applied as a module on top of KgCoOp (→ DeKg\(_{\text{KgCoOp}}\)) and TCP (→ DeKg\(_{\text{TCP}}\)), adding zero extra parameters beyond the TCP embedding module.

Key Contributions

  • Identifies pre-training bias as a fundamental failure mode of KGCO methods and frames it as a consistency–divergence contradiction.
  • Proposes HSIC-based independence regularization (\(\mathcal{L}_{kd}\)) that adds zero parameters to the model and complements the existing consistency loss.
  • Provides a plug-and-play module compatible with any embedding-aligned KGCO method (demonstrated on KgCoOp and TCP).
  • Shows that \(\mathcal{L}_{kd}\) implicitly enforces inter-class divergence among learnable prompts, improving discriminability on base classes while retaining generalization on new classes.

Results

Base-to-new generalization (16-shot, ViT-B/16, averaged over 11 datasets): - DeKg\(_{\text{TCP}}\): Base 84.96% / New 76.38% / H 80.44 vs. TCP: 84.13 / 75.36 / 79.51 - DeKg\(_{\text{KgCoOp}}\): H 78.57 vs. KgCoOp H 77.00 (+1.57%), with no added parameters - DeKg\(_{\text{TCP}}\) outperforms PromptSRC (H 79.97) and TCP (H 79.51) on harmonic mean

Cross-dataset generalization (train on ImageNet, test on 10 datasets): - DeKg\(_{\text{TCP}}\): avg 66.64% vs. TCP 66.29%, with +1.60% on Aircraft (25.05% vs. 23.45%)

Few-shot (4-shot, 11 datasets): - DeKg\(_{\text{TCP}}\): avg 77.06% vs. TCP 76.72% (+0.34%); DeKg\(_{\text{KgCoOp}}\): 75.12% vs. KgCoOp 74.48% (+0.64%)

Ablation (base-to-new with KgCoOp backbone): - CoOp alone: H 74.60; +\(\mathcal{L}_{kg}\) only: H 77.00; +\(\mathcal{L}_{kd}\) only: H 75.93; both: H 78.57

Limitations

  • Evaluated only on classification tasks with ViT-B/16; generalization to other VLM architectures (e.g., LLaVA-style models) or generation tasks is untested.
  • DeKg\(_{\text{KgCoOp}}\) underperforms PromptSRC and TCP on cross-dataset generalization (65.33% vs. 65.81% / 66.29%), suggesting the baseline's capacity matters.
  • Incompatible with gradient-aligned methods (ProGrad) and multi-modal prompt methods (PromptSRC) without modification, limiting plug-and-play scope.
  • Hyperparameter ratio \(\lambda/\mu\) requires grid search; sensitivity analysis shows performance degrades noticeably outside the optimal ratio.
  • No evaluation beyond 16-shot for base-to-new; behavior with very large shot counts is unknown.

Relevance to Vision-Language Models

DeKg directly targets prompt tuning for CLIP-style VLMs—one of the dominant adaptation paradigms—and resolves a concrete failure mode (pre-training bias) in the widely-used KGCO family. For researchers tracking VLMs, it demonstrates that the consistency–diversity trade-off in soft prompt learning can be precisely controlled via kernel-based independence measures rather than heuristic diversity terms. The HSIC regularization is theoretically clean, parameter-free, and orthogonal to other prompt learning innovations (class-specific knowledge, self-ensembling), making it a readily composable component for future VLM adaptation methods.