Skip to content

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

🕒 Published (v1): 2026-07-04 14:45 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; unified self-ensembling for test-time prompt tuning advances CLIP-based TTA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

USE revisits Test-Time Prompt Tuning (TPT) for CLIP and reveals that its marginal entropy minimization is equivalent to reverse cross-entropy against a self-generated pseudo-label. Building on this, USE introduces a Self-Ensembling (SE) strategy that adaptively weights the original test image against its augmented views, and applies the same strategy at both optimization and inference to ensure stage-level consistency.

Problem

TPT minimizes marginal entropy over multiple augmented views but (1) implicitly underweights or excludes the original (weak) test image when constructing the pseudo-label, and (2) creates a stage inconsistency: optimization averages across selected augmented views, while inference predicts only on the single weak augmentation \(A_0(x)\). Prior work neither recognized the pseudo-label interpretation nor aligned the two stages.

Method

Pseudo-label reinterpretation. By inserting an auxiliary distribution \(q\) into the marginal entropy: $\(L_{\text{mem}} = \underbrace{-\bar{p}_\theta(x)\log q}_{\text{RCE}} + \underbrace{KL(\bar{p}_\theta(x)\,\|\,q)}_{\text{already optimal if } q=\text{stopgrad}(\bar{p})}\)$ the KL term vanishes, leaving \(L_{\text{mem}} \approx \text{RCE}(\bar{p}_\theta(x); q(x))\), which drives augmented views toward a fixed pseudo-label \(q\).

Self-Ensembling (SE). To improve pseudo-label quality, SE computes the relative entropy rank of the weak augmentation \(A_0(x)\) among the \(N{-}1\) strong augmentations: $\(\delta = \frac{1}{N-1}\sum_{i=1}^{N-1} \mathbf{1}\!\left[H(p_\theta(A_i(x))) > H(p_\theta(A_0(x)))\right]\)$ This is rescaled to \(\beta = 0.5 + \gamma(\delta - 0.5)\) with \(\gamma=0.4\) (constraining \(\beta \in [0.3, 0.7]\)), and the pseudo-label becomes: $\(q(x) = \beta \cdot p_w(x) + (1-\beta)\cdot p_s(x)\)$ where \(p_w\) is the weak prediction and \(p_s\) averages selected confident strong-augmentation predictions.

Unified Framework (USE). The RCE loss is minimized over the full selected set \(S\). At inference, the same SE formula is re-applied with the updated model to produce \(\hat{p}(x) = \beta \cdot p_w(x) + (1-\beta)\cdot p_s(x)\), making optimization and inference objectives identical. A skip technique disables optimization when all predictions in \(S\) are already consistent, reducing cost. SE alone can serve as an optimization-free TTA method by applying the ensemble directly without any prompt update.

Key Contributions

  • Decomposition of TPT's marginal entropy minimization into RCE + KL terms, revealing implicit pseudo-label learning.
  • SE: a lightweight, optimization-free TTA method based on entropy-adaptive weak/strong augmentation ensembling.
  • USE: a unified framework that eliminates the train–inference objective inconsistency in TPT and its successors.
  • SE is a plug-in module that integrates with existing TTA methods (TPT, C-TPT, R-TPT, STS) without significant overhead.

Results

  • ImageNet + OOD variants (ViT-B/16): USE achieves 64.18% average vs. TPT 62.37%, STS 63.89% (best optimization-based); SE (optimization-free) reaches 63.65% vs. ZERO 63.80% (second among optimization-free).
  • Fine-grained datasets (ViT-B/16, 10 datasets): USE 65.35% avg vs. TPT 64.84%; SE 65.02% avg vs. MTA 64.77%.
  • ResNet-50 (OOD): SE achieves 47.80% avg, surpassing most optimization-based methods (TPT: 47.19%).
  • EuroSAT (ViT-B/16): SE reaches 44.44% (+2.39% over zero-shot CLIP), while ZERO degrades below zero-shot.
  • CoOp initialization (ViT-B/16, fine-grained): SE attains 64.57% avg, outperforming all counterparts; USE matches best-performing RLCF at lower computational cost.
  • STS marginally outperforms USE on ImageNet-A alone; USE maintains superior averages across all other splits.

Limitations

  • \(\gamma = 0.4\) is fixed empirically across all experiments; sensitivity is deferred to an appendix and not fully analyzed in the main text.
  • Evaluated exclusively on CLIP (ViT-B/16 and ResNet-50); generalization to other VLM architectures (e.g., ALIGN, SigLIP) is not demonstrated.
  • SE outperforms or matches optimization-based methods on ResNet-50 but only marginally lags STS on ImageNet-A for ViT-B/16, suggesting the adaptive weighting does not uniformly dominate on adversarial hard examples.
  • The skip technique for computational savings is heuristic and may interact with datasets where most augmentations are conflicted.

Relevance to Vision-Language Models

USE directly addresses the core challenge of adapting CLIP-style VLMs to distribution-shifted test data without any labeled examples, a high-priority problem given VLMs' real-world deployment. The pseudo-label reinterpretation provides a cleaner theoretical grounding for the entire family of entropy-minimization TTA methods built atop CLIP, which is foundational for the field. The SE strategy is architecturally agnostic and plugs into any augmentation-based TTA framework, making it broadly applicable as the VLM ecosystem expands beyond CLIP. The unified train–inference objective principle established here is directly relevant to future work on test-time fine-tuning of multimodal foundation models.