Skip to content

A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; angular diversity calibration addresses prompt dispersion gap in VLM TTA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A-TPT is a test-time prompt tuning framework for CLIP-like VLMs that improves calibration by maximizing the minimum pairwise angular distance between normalized class-wise textual features on the unit hypersphere. It addresses a fundamental flaw in prior calibration methods (L2 dispersion and orthogonality constraints) that fail to guarantee uniform feature separation, especially when the number of classes exceeds the embedding dimension. A-TPT consistently reduces Expected Calibration Error (ECE) while maintaining competitive accuracy across diverse datasets.

Problem

Test-time prompt tuning (TPT) boosts zero-shot accuracy in VLMs but worsens calibration — predicted confidence diverges from true accuracy, producing overconfident predictions. Prior calibration remedies either (1) maximize average L2 dispersion from a centroid (C-TPT/ATFD), which can still leave features clustered, or (2) enforce orthogonality (O-TPT), which fails when \(N > |D|\) (e.g., 1000 ImageNet classes in a 512-d CLIP space, making pairwise orthogonality impossible) and underutilizes hyperspherical space when \(N < |D|\). Neither method guarantees uniform angular separation across the hypersphere.

Method

A-TPT introduces an angular diversity regularization term into the TPT objective. Given the set of class-conditioned textual features \(\{t_k\}_{k=1}^N\) produced by learnable prompt vectors, A-TPT maximizes the minimum pairwise angular distance between their \(\ell_2\)-normalized counterparts on the unit hypersphere — directly solving the Tammes (best-packing) problem:

\[\mathcal{L}_{\text{A-TPT}} = -\min_{i \neq j} \angle(t_i, t_j)\]

This is minimized jointly with the entropy-minimization loss from standard TPT via a scalar \(\lambda\) that controls regularization strength. By penalizing the closest pair of prompt directions at each gradient step, A-TPT pushes all class features toward a maximally spread, uniform configuration on the hypersphere regardless of whether \(N > |D|\) or \(N < |D|\). No labeled data is required; optimization runs at inference time over augmented views of the single test image.

Key Contributions

  • A numerical optimization method (A-TPT) that maximizes the minimum pairwise angular distance between normalized textual features during test-time prompt tuning, unifying both \(N > |D|\) and \(N < |D|\) regimes.
  • Theoretical and empirical grounding for why angular diversity (not L2 dispersion or orthogonality) is the right objective for VLM calibration — framed via the Tammes problem and hyperspherical uniformity.
  • Demonstration that A-TPT achieves lower aggregate ECE than C-TPT and O-TPT across multiple backbones, datasets (including medical), and natural distribution shifts, while maintaining comparable accuracy.

Results

  • Overall ECE (CLIP ViT-B/16): A-TPT 3.26 vs. O-TPT 4.36, C-TPT 5.42, TPT 11.25, Baseline 4.40.
  • \(N > |D|\) group ECE: A-TPT 2.92 vs. O-TPT 4.27, C-TPT 5.58, TPT 12.60.
  • \(N < |D|\) group ECE: A-TPT 3.60 vs. O-TPT 4.44, C-TPT 5.25, TPT 9.89.
  • Superior calibration on natural distribution shifts (ImageNet variants) and generalization to medical imaging benchmarks.
  • Accuracy roughly preserved: A-TPT 61.27% vs. O-TPT 61.17%, C-TPT 62.07% overall (slight accuracy trade-off for significantly lower ECE).

Limitations

  • Accuracy slightly decreases relative to C-TPT (62.07% → 61.27%), indicating a calibration–accuracy trade-off.
  • The Tammes/best-packing optimization introduces an additional hyperparameter \(\lambda\); sensitivity analysis is not fully explored in the main text.
  • Evaluation is primarily on CLIP; generalization to other VLM architectures (e.g., LLaVA, BLIP-2) is not demonstrated.
  • Medical dataset experiments are presented as additional evidence but without detailed analysis of clinical significance or dataset diversity.

Relevance to Vision-Language Models

Calibration is an underexplored failure mode of prompt-tuned VLMs: achieving high accuracy without reliable confidence is dangerous for safety-critical deployments. A-TPT directly addresses this for the CLIP family by showing that the geometry of the text embedding space — specifically, uniform angular coverage of the hypersphere — governs calibration quality. This connects to broader work on contrastive representation geometry (Wang & Isola 2020) and extends the TPT literature (Shu 2022, CoOp, CoCoOp) toward trustworthy zero-shot adaptation. The result that \(N > |D|\) is a systematic failure mode for orthogonality-based methods is a practically important finding given that large-scale recognition tasks (ImageNet-1k, 512-d CLIP) routinely sit in this regime.