Skip to content

Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AsymVLM is a post-hoc probabilistic adapter for frozen VLMs (e.g., CLIP) that exploits the structurally asymmetric uncertainty between text and image modalities: text embeddings are mapped to directional distributions (vMF or power spherical) on the unit hypersphere, while image embeddings remain deterministic. This asymmetric design improves both uncertainty quantification and cross-modal retrieval without retraining the backbone.

Problem

Deterministic VLMs (CLIP, BLIP, SigLIP) produce point-estimate embeddings that cannot represent aleatoric ambiguity inherent in language (one text concept → many visual instances) versus images (one image → many valid captions). Existing post-hoc probabilistic adapters (ProbVLM, BayesVLM) apply symmetric probabilistic treatment to both modalities and operate in Euclidean space, ignoring (1) the structural asymmetry between text and image uncertainty and (2) the fact that pretrained VLM embeddings live on the unit hypersphere S^{d-1}.

Method

AsymVLM trains a lightweight text adapter g_T on top of a frozen text encoder while leaving the image encoder fully deterministic. The adapter outputs parameters (µ, κ) of a directional distribution — either von Mises-Fisher (vMF) or power spherical (PS) — on S^{d-1}. The log-likelihood matrix L(m,n) = ln p_{z^T_m}(z^I_n) is optimized with an InfoNCE objective, where the vMF kernel becomes κ(t_r)·µ(t_r)^⊤ z^I_s + F_d(κ(t_r)) (a tractable approximation to the intractable Bessel normalizer). This is provably a monotone extension of CLIP's cosine-similarity loss with concentration parameter κ encoding text uncertainty. Uncertainty is quantified as u(t) = 1/κ(t).

Key Contributions

  • Formal asymmetry analysis: text-to-image mapping is one-to-many due to abstraction + visual variability (high aleatoric); image-to-text is one-to-many due to descriptive variance (lower aleatoric, epistemic).
  • AsymVLM adapter: probabilistic text embeddings via vMF/PS on the hypersphere; deterministic image embeddings retained.
  • Derivation showing AsymVLM is a strict generalization of CLIP's InfoNCE loss, adding uncertainty calibration without departing from cosine similarity semantics.
  • Ablation isolating (i) asymmetric architecture necessity and (ii) hyperspherical vs. Euclidean (Gaussian) distribution choice.
  • Demonstrations of uncertainty-aware "none-of-the-above" rejection and improved robust fine-tuning against catastrophic forgetting.

Results

  • Cross-modal retrieval (Recall@1, T2I / I2T) on CLIP ViT-B/32 backbone:
  • MS-COCO: AsymVLM vMF 0.561 / 0.392 vs. best baseline BayesVLM 0.506 / 0.323
  • Flickr-30k: AsymVLM vMF 0.688 / 0.504 vs. best baseline PFE* 0.680 / 0.451
  • CC-200k: AsymVLM vMF 0.395 / 0.383 vs. best baseline ProLIP 0.351 / 0.335
  • UQ quality: AsymVLM is the only method achieving strictly monotone Recall@1 decrease with uncertainty level across all datasets; achieves R² ≥ 0.990 and Spearman S = −1.000 on CC-200k.
  • None-of-the-above: AsymVLM vMF reaches 0.857 positive accuracy and 0.587 negative accuracy vs. CLIP 0.888 / 0.009 and margin-based 0.584 / 0.579.
  • Robust fine-tuning (zero-shot transfer after fine-tuning on MS-COCO): AsymVLM vMF 0.837 / 0.477 / 0.940 / 0.507 on CIFAR-10 / CIFAR-100 / STL-10 / ImageNet-1k vs. Determ.FT 0.684 / 0.300 / 0.829 / 0.395.
  • HierarCap analysis confirms uncertainty decreases monotonically from abstract (Level 0) to detailed (Level 3) captions.

Limitations

  • Only text uncertainty is modeled; image embeddings remain deterministic. Image uncertainty would require data augmentation or alternative strategies outside the current scope.
  • Evaluated exclusively on CLIP ViT-B/32; transferability to larger backbones (ViT-L, SigLIP, BLIP-2) not empirically verified in the main paper.
  • Downstream application coverage is limited to two illustrative tasks; broader evaluation (e.g., VQA, grounding, medical imaging) is left to future work.

Relevance to Vision-Language Models

AsymVLM directly addresses a known weakness of contrastive VLMs: their inability to express embedding uncertainty, which undermines reliability in ambiguous or open-vocabulary settings. By deriving a theoretically grounded post-hoc adapter that respects the hyperspherical geometry of CLIP-family embeddings and the modality asymmetry, it provides a principled path to uncertainty-aware retrieval and classification without expensive retraining. This connects to growing interest in calibration, robustness, and safe deployment of VLMs, and complements work on probabilistic contrastive learning (PCME, PFE) and Bayesian adaptation (BayesVLM, ProbVLM) by introducing asymmetry as a first-class design criterion.