Skip to content

DualCnst: Enhancing Zero-Shot Out-of-Distribution Detection via Text-Image Consistency in Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DualCnst is a training-free, zero-shot OOD detection framework that augments CLIP's text-based semantic similarity with visual similarity computed against Stable Diffusion-synthesized exemplars for both ID and OOD labels. By fusing multi-layer image-to-image and image-to-text cosine similarities via a weighted sum-softmax score, it achieves state-of-the-art performance without requiring access to real in-distribution images.

Problem

Existing VLM-based zero-shot OOD detectors (e.g., MCM, NegLabel) rely exclusively on semantic similarity between test images and textual class labels. Semantically ambiguous OOD samples—those sharing concept-level overlap with ID classes (e.g., "zebra" vs. "wild horse")—cause significant ID/OOD score distribution overlap that text-only scoring cannot resolve, even with expanded negative label spaces.

Method

DualCnst operates in two stages:

  1. Synthesis: Given ID labels \(Y^{id}\), NegLabel mines OOD labels \(Y^{ood}\). Stable Diffusion XL-Turbo generates one synthetic image per label from the combined space \(Y^{id} \cup Y^{ood}\) using a standard prompt "A photo of a \<label>".

  2. Dual-consistency scoring: For a test image \(x\), two signals are computed:

  3. Image-to-text similarity \(s_{i,\text{text}}(x)\): cosine similarity between CLIP image embedding and each label's text embedding.
  4. Image-to-image similarity \(s_{i,\text{img}}(x)\): weighted sum of cosine similarities between \(x\)'s intermediate ViT features (layers 3, 6, 9, 12 for ViT-B/16, weight 0.25 each) and the corresponding synthetic image embeddings.

These are fused as \(\tilde{s}_i(x) = \alpha \cdot s_{i,\text{img}} + (1-\alpha) \cdot s_{i,\text{text}}\) with \(\alpha=0.2\), then aggregated into a softmax-based ID score \(S_{\text{DualCnst}}(x)\) summed over ID indices. Detection threshold \(\lambda\) is set at 95% TPR.

A theoretical analysis via CLT shows that augmenting the negative label space with \(N\) multimodal signals reduces score variance by a factor \(N'(N) = N(N-1)/(a^2 + N - 2a)\), monotonically decreasing FPR95 as \(N\) grows.

Key Contributions

  • Dual-consistency scoring framework that combines CLIP semantic similarity with multi-layer visual similarity to synthesized exemplars for zero-shot OOD detection.
  • Text-driven image synthesis pipeline using Stable Diffusion to build visual class prototypes for both ID and mined OOD categories without any real ID images.
  • Theoretical proof that multimodal negative label aggregation reduces similarity score variance and improves ID–OOD separability.
  • Plug-and-play design: compatible with any prior text-only VLM OOD method (e.g., NegLabel as the label-mining backbone); all synthetic images are pre-generated offline, adding no inference-time overhead.

Results

  • Far-OOD (ImageNet-1k): DualCnst achieves average FPR95 of 21.45% and AUROC of 94.72%, reducing FPR95 by 3.95% absolute vs. NegLabel (25.40%) and improving AUROC by 0.51%.
  • iNaturalist: FPR95 0.98% vs. NegLabel 1.91%; AUROC 99.70% vs. 99.49%.
  • Textures: FPR95 34.91% vs. NegLabel 43.56%; AUROC 91.73% vs. 90.22%.
  • Near-OOD (ImageNet-10/20): Average FPR95 7.45% vs. NegLabel 11.35%; AUROC 98.20% vs. 97.95%.
  • ImageNet-10 as ID: FPR95 2.20% vs. NegLabel 5.10%.
  • Fine-grained IDs (Stanford-Cars, Oxford-Pet, CUB-200, Food-101): Near-zero FPR95 and AUROC ≈ 100% across all four OOD benchmarks, consistently matching or exceeding NegLabel.
  • ImageNet-100: FPR95 12.83% vs. NegLabel 14.05%; AUROC 97.60% vs. 97.36%.
  • T-SNE visualizations confirm significantly better ID/OOD cluster separation vs. NegLabel on semantically challenging pairs (horse/zebra, warplane/space shuttle).

Limitations

  • Requires running Stable Diffusion to pre-generate one synthetic image per class label; this is a one-time cost but introduces a dependency on a generative model and storage.
  • Sensitive to the quality of mined OOD labels (relies on NegLabel); if negative label mining fails, visual synthesis may not cover the relevant OOD manifold.
  • Hyperparameters \(\alpha\) and layer weights \(w\) require validation on ID-only data; generalization across all dataset granularities is assumed but not proven for truly novel domains.
  • Theoretical analysis assumes i.i.d. similarity scores across modalities—an idealization that may not hold when synthetic images are visually correlated.
  • Evaluation uses a single generated image per label; diversity of synthesized exemplars and the effect of random seed variation is acknowledged but only partially ablated.

Relevance to Vision-Language Models

DualCnst directly addresses a known failure mode of CLIP-based VLMs: semantic similarity alone is insufficient to discriminate visually distinct but semantically proximate classes, which is a fundamental constraint of contrastive text-image pretraining. By closing the loop between CLIP's text-image embedding space and generative vision models (Stable Diffusion), the work demonstrates how text-conditioned image synthesis can serve as a practical proxy for real ID data, extending zero-shot capabilities without fine-tuning. This is directly relevant to researchers tracking VLMs as it exemplifies how generative augmentation can compensate for the modality gap in retrieval- and similarity-based VLM pipelines. The plug-and-play design makes it applicable to any CLIP-style backbone, and the theoretical grounding in score-variance reduction offers a principled justification for multimodal label expansion strategies.