Skip to content

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining

🕒 Published (v1): 2026-04-30 11:01 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Flatness-aware pretraining improves calibration in VLM test-time prompt tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Test-time prompt tuning (TPT) for CLIP-based VLMs produces poorly calibrated models. This paper reveals that existing calibration regularizers implicitly push prompts toward flat loss-landscape minima, then proposes Flatness-aware Prompt Pretraining (FPP): a data-free offline step that initializes prompts in flatter regions before any test-time adaptation, improving both calibration and accuracy without modifying the TPT pipeline itself.

Problem

TPT methods adapt CLIP prompts at inference via entropy minimization, but consistently produce overconfident, poorly calibrated predictions. Prior fixes (C-TPT, O-TPT) add geometric regularization losses that improve calibration but degrade accuracy, and the mechanism by which calibration improves was not understood. No prior work connects loss-landscape sharpness to calibration quality in the prompt-tuning setting.

Method

FPP (Flatness-aware Prompt Pretraining) is an offline pretraining step applied once per dataset/class-set, before any test-time adaptation.

Theoretical insight first: The authors prove that the regularization losses of C-TPT and O-TPT satisfy \(\mathcal{H}(T) = \alpha L_\text{reg}(T) + \beta + O(D^{-3/2})\), meaning increasing regularization loss raises expected entropy — equivalent to a SAM-like perturbation that steers optimization toward flat minima.

FPP objective: Two complementary losses are jointly minimized:

\[\mathcal{L}_\text{FPP} = \mathcal{L}_\text{align} + \lambda \mathcal{L}_\text{flat}\]
  • Flatness loss \(\mathcal{L}_\text{flat} = \text{dist}_\text{cos}\!\bigl(f_T(C{+}\varepsilon_1;\,\theta{+}\varepsilon_2),\, f_T(C;\,\theta)\bigr)\): penalizes sensitivity of text features to small random perturbations \(\varepsilon_1, \varepsilon_2 \sim \mathcal{N}(0,\sigma^2 I)\), provably reducing sharpness for any differentiable loss.
  • Alignment loss \(\mathcal{L}_\text{align} = \text{dist}_\text{L2}\!\bigl(f_T(C;\,\theta),\, f_T(C;\,\theta_0^\text{zs})\bigr)\): keeps learned text features close to the zero-shot prompt \(\theta_0^\text{zs}\), preserving semantic structure and zero-shot accuracy.
  • \(\lambda = \frac{\gamma_1 + \gamma_2}{K}\) downweights flatness for larger class sets where alignment is harder to maintain.

FPP depends only on class names \(C\) and \(\theta_0^\text{zs}\) — no images required. The resulting \(\theta_0^\text{FPP}\) replaces the standard initialization in any TPT pipeline; all subsequent adaptation steps remain unchanged.

Key Contributions

  • Formal proof that C-TPT and O-TPT regularization losses implicitly act as SAM-like perturbations, linking text-feature dispersion to flat-minima convergence.
  • Empirical demonstration that prompts in flatter loss-landscape regions achieve substantially lower ECE/SCE, establishing sharpness as a key calibration factor in TPT.
  • FPP: a data-free, offline prompt pretraining framework that initializes prompts in flat regions; zero additional cost at test time and plug-and-play with existing TPT methods.
  • First method to simultaneously achieve SOTA accuracy and SOTA calibration (ECE + SCE) in TPT — prior methods traded one for the other.

Results

All results use CLIP-ViT-B/16; metrics are Accuracy (Acc.), Expected Calibration Error (ECE), and Static Calibration Error (SCE) averaged over 10 fine-grained classification datasets.

Hard-prompt setting (Table 1): - FPP: Avg Acc. 65.37%, Avg ECE 4.13%, Avg SCE 0.96% - vs. TPT (NeurIPS 2022): 64.62% / 11.67% / 1.15% - vs. C-TPT (ICLR 2024): 64.48% / 5.32% / 1.11% - vs. O-TPT (CVPR 2025): 64.12% / 4.46% / 1.15% - FPP is the only method achieving SOTA in all three metrics simultaneously.

Supervised-prompt initialization (Table 2): - MaPLe+FPP: Avg Acc. 67.01%, Avg ECE 4.07% — best accuracy, competitive ECE - CoOp+FPP: Avg Acc. 64.78%, Avg ECE 5.67% — best accuracy among CoOp variants; larger improvement margin than hard-prompt setting.

DynaPrompt framework (Table 3): - DynaPrompt+FPP: Avg Acc. 65.26%, Avg ECE 4.56%, Avg SCE 1.13% — SOTA across all metrics vs. DynaPrompt+C-TPT and DynaPrompt+O-TPT.

OOD setting: FPP achieves strong ECE/SCE improvements with only −0.42% accuracy vs. TPT, whereas O-TPT drops accuracy by >2%.

Limitations

  • Evaluated exclusively on CLIP-ViT-B/16; generalization to other VLM backbones (e.g., larger ViT variants, ALIGN, BLIP) is not demonstrated.
  • FPP pretraining must be re-run for each new class-name set \(C\), which may be costly for tasks with very large or dynamic label spaces.
  • The flatness–calibration connection is established empirically and theoretically only under the uniform-sphere assumption for image features (Theorem 1), which may not hold in practice.
  • Performance on SAT and Aircrafts datasets remains volatile (e.g., FPP SAT ECE = 5.19% still lags zero-shot CLIP's 7.40% but underperforms on some ECE comparisons per dataset), suggesting dataset-specific sensitivity.
  • No evaluation on out-of-distribution generalization benchmarks beyond the fine-grained datasets (e.g., ImageNet-A/R/Sketch).

Relevance to Vision-Language Models

This work directly addresses a core reliability concern in deploying CLIP-style VLMs: test-time-adapted models are systematically overconfident, undermining their use in safety-critical settings. The key insight — that sharpness of the prompt's loss landscape governs calibration quality — provides a principled geometric explanation for why ad-hoc output-space regularizers work, and points toward a cleaner fix via initialization rather than runtime constraints. For researchers tracking VLMs, FPP is notable because it achieves calibration improvements as a preprocessing step with zero inference overhead and no labeled data, making it immediately composable with any entropy-minimization TPT variant. The flatness–calibration link also connects the VLM prompt-tuning literature to the broader flat-minima generalization literature (SAM and variants), opening a new research axis for prompt optimization.