Skip to content

An Information-theoretical Framework for Understanding Out-of-distribution Detection with Pretrained Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper provides a formal information-theoretic justification for why CLIP-based post-hoc OOD detectors work, showing that MCM and NegLabel are both stochastic Monte Carlo estimators of point-wise mutual information (PMI) within an energy-based model framework. Motivated by divide-and-conquer, the authors decompose PMI via the chain rule into sub-terms estimated over image sub-views, provably increasing the estimation upper bound and reducing underestimation bias. The resulting method achieves state-of-the-art zero-shot OOD detection on ImageNet-1K benchmarks.

Problem

CLIP-based post-hoc OOD detectors (MCM, NegLabel) have demonstrated strong empirical performance but lack rigorous theoretical grounding — there is no formal explanation of why they work, what they are implicitly computing, or how to principally improve upon them. Prior theoretical OOD work does not extend to the multi-modal VLM setting.

Method

The paper models in-distribution (ID) data via an energy-based model where energy functions are defined as PMI between input image \(x\) and each ID label \(y\): \(E(x) = \frac{1}{\alpha}\log\sum_{y\in\mathcal{Y}_I}\exp[\alpha W(x,y)]\). Since PMI is intractable, CLIP's softmax output is used as a variational estimator of \(p_{Y|X}\). The authors prove (Theorems 1–2) that MCM and NegLabel are Monte Carlo estimators of this energy with different upper bounds (\(\log K\) and \(\log(K+L/T)\) nats, respectively).

For the algorithmic contribution, they apply the chain rule on PMI: \(W(x,y) = W(\tilde{x},y) + W(x,y|\tilde{x})\), where \(\tilde{x}\) is a sub-view of \(x\) generated by random cropping. Each term is estimated separately using CLIP: \(W(\tilde{x},y)\) via the NegLabel-style softmax over negative labels on the cropped view, and \(W(x,y|\tilde{x})\) via a conditioned critic \(h_\theta(x,\tilde{x},y) = r^\top[\beta\tilde{z}+(1-\beta)z]/\kappa\) that blends original and sub-view features. The combined scorer \(S_\text{ours}\) achieves an upper bound of \(\log(K+L)\) nats (Theorem 4), strictly higher than NegLabel's \(\log(K+L/T)\) without requiring \(TL\) negative labels. Final scores are averaged over \(V=2\) random crops.

Key Contributions

  • Information-theoretic framework unifying CLIP-based OOD detectors: both MCM and NegLabel are shown to be Monte Carlo PMI estimators within an energy-based ID density model.
  • Proof that negative labels raise the PMI estimation upper bound (\(\log K \to \log(K+L/T)\)) and that grouping reduces variance by approximating the expectation via multi-sample averaging.
  • Chain-rule PMI decomposition over image sub-views that provably raises the upper bound to \(\log(K+L)\) without the constraint of large \(T\) required by NegLabel.
  • State-of-the-art zero-shot OOD detection on ImageNet-1K (iNaturalist, SUN, Places365, Textures) using CLIP ViT-B/16, training-free.

Results

  • vs. NegLabel (Table 1, ImageNet-1K, average): Ours (Median) achieves AUROC 95.07% vs. NegLabel 94.21% (+0.86%) and FPR95 21.20% vs. 25.40% (−4.20 pp); Ours (Mean) achieves AUROC 96.00%, FPR95 22.36%.
  • Paper's stated improvement: 3.16 pp reduction in FPR95 and 2.21 pp gain in AUROC averaged across four OOD datasets versus NegLabel.
  • iNaturalist (best single dataset): Ours (Median) AUROC 99.70%, FPR95 1.04% vs. NegLabel 99.49%/1.91%.
  • Domain-generalizable OOD (Table 2): On ImageNet-S and ImageNet-A as shifted ID sets, Ours consistently outperforms NegLabel on all four OOD datasets.
  • With learned prompts (Table 3): Ours+AdaNeg outperforms NegLabel+AdaNeg on average FPR95 under both NegPro and LAPT prompt settings.
  • Outperforms all training-free zero-shot baselines (ZOC, MCM, NegLabel) and most training-based methods (LoCoOp, LAPT, NegPro) on average metrics.

Limitations

  • Theoretical guarantees require the CLIP critic \(h_\theta\) to satisfy log-linear optimality assumptions (Theorem 4 conditions), which may not hold in practice.
  • Random cropping introduces variance; the paper mitigates this by averaging \(V=2\) crops but aggressive cropping (\(\lambda\) too small) degrades semantics and performance.
  • Method uses the same NegMining pipeline as NegLabel (top 15% WordNet words dissimilar from ID labels), inheriting its sensitivity to the negative label source.
  • NegLabel's known degradation under excessive negative labels is avoided by the decomposition, but the mechanism for choosing \(L\) and \(T\) is inherited heuristically.
  • Evaluation uses CLIP ViT-B/16 only; scalability to larger CLIP variants or other VLM architectures (e.g., SigLIP, ALIGN) is not demonstrated.

Relevance to Vision-Language Models

This work directly addresses the theoretical foundations of using CLIP for zero-shot OOD detection, a core reliability concern for VLM deployment in open-world settings. By proving that CLIP's contrastive text-image alignment implicitly enables PMI-based density estimation, it explains why multi-modal post-hoc detectors outperform unimodal ones without any fine-tuning. The chain-rule decomposition via sub-views introduces a new inductive bias for exploiting CLIP's feature space that is architecturally agnostic, making it applicable to any CLIP-family model. Researchers tracking VLMs will find this paper essential for understanding the theoretical guarantees of prompt-based OOD methods and as a principled basis for future extensions (e.g., combining with visual negative proxies via AdaNeg).