Skip to content

Post-hoc Probabilistic Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; post-hoc probabilistic VLMs with calibrated uncertainty in joint space

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BayesVLM applies a Laplace approximation post-hoc to the linear projection layers of pre-trained VLMs (CLIP, SigLIP) to obtain principled Bayesian uncertainty estimates over cosine similarities without retraining. The method derives an analytical Gaussian approximation (ProbCosine) for the distribution over cosine similarity, enabling efficient uncertainty propagation. It demonstrates improved calibration over vanilla CLIP and enables uncertainty-driven active fine-tuning with substantial gains over deterministic baselines.

Problem

Standard VLMs (CLIP, SigLIP) produce deterministic embeddings that yield overconfident predictions under domain shift. Existing uncertainty approaches either require costly retraining/fine-tuning, architectural changes (probabilistic adapters), or only address calibration without capturing epistemic uncertainty. No prior work provides a training-free, model-agnostic, post-hoc Bayesian formulation directly applicable to the contrastive VLM objective.

Method

BayesVLM applies the Laplace approximation (LA) to the linear projection layers \(P \in \mathbb{R}^{d \times d_\text{IMG}}\) and \(Q \in \mathbb{R}^{d \times d_\text{TXT}}\) of a frozen VLM:

  1. Likelihood decomposition: The InfoNCE loss entangles modalities and violates i.i.d. assumptions. BayesVLM factorizes it into per-modality conditionals (\(p(x_i^\text{IMG} \mid X^\text{TXT}, \theta)\) and \(p(x_i^\text{TXT} \mid X^\text{IMG}, \theta)\)), enabling separate posterior inference on \(P\) and \(Q\) with \(P \perp\!\!\!\perp Q\).

  2. Hessian approximation: The full Hessian is intractable; the Kronecker-factored GGN (KFAC) is used, yielding: $\(\Sigma_\text{IMG} \approx \left(\sqrt{\tau}\tilde{A}_\text{IMG}^{-1}\right) \otimes \left(\sqrt{\tau}\tilde{B}_\text{IMG}^{-1}\right)\)$ Kronecker factors \(A, B\) are computed on a calibration subset with pseudo-count \(\tau\).

  3. Probabilistic embeddings: The Gaussian posterior over \(P\) propagates analytically through the linear projection: $\(p(g \mid \mathcal{D}) = \mathcal{N}\!\left(P_\text{MAP}\phi(x^\text{IMG}),\; \phi(x^\text{IMG})^\top \tilde{A}_\text{IMG}^{-1} \phi(x^\text{IMG})\, \tilde{B}_\text{IMG}^{-1}\right)\)$

  4. ProbCosine: A closed-form Gaussian approximation of the cosine similarity distribution using first two moments of \(g\) and \(h\): $\(\mathbb{E}[S_\text{COS}(g,h)] \approx \frac{\sum_i \mu_{g,i}\mu_{h,i}}{\sqrt{\sum_i \mu_{g,i}^2 + \sigma_{g,i}^2}\,\sqrt{\sum_i \mu_{h,i}^2 + \sigma_{h,i}^2}}\)$ with a corresponding variance expression, then a probit approximation for class posteriors.

  5. Active learning: BayesVLM drives BALD and EPIG acquisition functions; an online LA updates \(P\) and Kronecker factors after each labeled sample is acquired.

Key Contributions

  • First direct Bayesian formulation of VLMs with an analytical distribution over cosine similarities (ProbCosine), enabling no-overhead uncertainty propagation.
  • Post-hoc LA applied only to projection layers \(P, Q\) — no retraining, no architecture changes, compatible with any VLM backbone including closed-source pre-training data.
  • Per-modality i.i.d. factorization of the InfoNCE/SigLIP losses that preserves cross-modal interactions via conditional independence at the MAP estimate.
  • Demonstrated efficacy on zero-shot calibration benchmarks and Bayesian active few-shot learning with BALD/EPIG acquisition functions.

Results

Zero-shot calibration (OpenCLIP ViT-B-32): - BayesVLM matches or exceeds CLIP accuracy (e.g., 93.62% vs. 93.61% on CIFAR-10) while substantially reducing ECE: 0.72 vs. 1.45 (CLIP) vs. 1.88 (temp. scaling) on CIFAR-10. - ECE improvements are consistent across all 7 benchmarks; e.g., Flowers-102: 4.22 (BayesVLM) vs. 6.59 (CLIP) vs. 5.51 (temp. scaling). - NLPD competitive with or better than temperature scaling on most datasets.

ProbCosine on probabilistic VLMs (PCME++): - Applying ProbCosine to PCME++ consistently improves ECE across all 7 benchmarks; e.g., Flowers-102 ECE drops from 8.81 to 2.79, SUN397 from 26.03 to 19.41. - NLPD also improves uniformly (e.g., SUN397: 3.83 → 3.50).

Active learning (EuroSAT, OfficeHome cross-domain): - BayesVLM with BALD/EPIG substantially reduces misclassification rate compared to fine-tuned vanilla CLIP after active support-set selection (qualitative in Fig. 2; quantitative in App.). - Reduces overconfident predictions in the pre-AL zero-shot setting (fewer high-confidence errors).

Limitations

  • Posterior is estimated only over linear projection layers; encoders (\(\phi\), \(\psi\)) remain deterministic, limiting epistemic uncertainty capture to the projection subspace.
  • The per-modality i.i.d. factorization and \(P \perp\!\!\!\perp Q\) assumption ignores cross-modal curvature (empirically found to be moderate but nonzero per App. F.13).
  • KFAC Kronecker factors must be estimated on a calibration subset; the pseudo-count \(\tau\) introduces a hyperparameter sensitive to subset size.
  • ProbCosine uses a moment-matched Gaussian approximation to a non-Gaussian distribution; approximation error increases in high-variance regimes.
  • Active learning evaluation is limited to relatively small cross-domain benchmarks (OfficeHome, ImageNet variants); scaling behavior to very large label pools is not assessed.

Relevance to Vision-Language Models

BayesVLM directly addresses a critical gap in deploying CLIP/SigLIP-class models: their deterministic embeddings yield overconfident predictions under distribution shift, which is particularly problematic for safety-critical or data-efficient applications. The method is notable because it retrofits Bayesian uncertainty onto the dominant contrastive VLM paradigm with no retraining cost, making it immediately applicable to the frozen backbone models widely used in zero-shot and transfer pipelines. The ProbCosine formulation is broadly applicable — it can pair with any model that produces distributional embeddings, including probabilistically pre-trained VLMs like PCME++/ProLIP. For researchers tracking VLMs, this opens a practical path to principled uncertainty-aware retrieval, classification, and active learning on top of existing large-scale checkpoints.