Skip to content

OOD-Barrier: Build a Middle-Barrier for Open-Set Single-Image Test Time Adaptation via Vision Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Open-IRT is a single-image test-time adaptation framework for open-set OOD detection that uses CLIP-based vision-language models to build a structured intermediate domain between in-distribution (ID) and out-of-distribution (OOD) feature clusters. It operates without batch data, adapting per-sample at inference time by combining polarity-aware prompt scoring with intermediate-domain contrastive learning. On CIFAR-100C→SVHN it achieves +8.45% AccHM, −10.80% FPR95, and +11.04% AUROC over prior state-of-the-art.

Problem

Existing test-time adaptation (TTA) methods assume either a closed-set setting or require batch processing, making them unsuitable for real-world single-image open-set streams where both ID and OOD samples arrive individually and unpredictably. VLM-based approaches like TPT fine-tune prompts but still assume closed-set inputs, while open-set batch methods (ROSITA, UniEnt) fail under single-sample constraints. The ID–OOD decision boundary degrades severely in such settings due to overlap in feature space.

Method

Open-IRT has two modules operating on a frozen CLIP text encoder and an SGD-updated vision encoder (batch size = 1):

Polarity-aware Prompt-based OOD Filter (PPF): For each incoming sample, cosine similarity is computed against positive prompts ("a photo of a [CLS]") and negative prompts ("a photo of no [CLS]"). A combined score S(f) = ϕ(sup_c[simpos + α·|simpos − simneg|]) captures both absolute semantic alignment and relative semantic polarity. Scores are accumulated in a sliding-window bank and fit with a Gaussian Mixture Model (GMM) to classify each sample as ID or OOD.

Intermediate Domain-based TTA (IDT): ID and OOD feature banks are maintained via FIFO queues. For each sample, an intermediate style feature fₘ is constructed by applying the statistics (mean/variance) of the opposite class (ID→OOD stats or OOD→ID stats) via instance normalization, placing fₘ between the two distributions in feature space. Two InfoNCE-based losses then push the sample feature away from this intermediate domain: Lmid uses KNN neighbors from the feature banks; Lpsd applies bidirectional cross-entropy pseudo-labels—positive CE for high-confidence ID predictions, negative CE (reverse gradient) for low-confidence samples. Total loss: L_TTA = Lpsd + Lmid.

Key Contributions

  • Semantic Contrast Hypothesis: Formalizes that ID samples have both high absolute similarity to positive prompts and a large polarity gap relative to negative prompts; this gap is exploited in the PPF scoring function.
  • Intermediate-Domain Hypothesis: Proves that increasing divergence from a constructed middle domain FM indirectly maximizes the Hilbert-Schmidt dissimilarity between ID and OOD distributions, circumventing the need to directly estimate the unknown OOD distribution.
  • PPF module: Dual-polarity VLM scoring + GMM classification for per-sample ID/OOD filtering without batch statistics.
  • IDT module: Style-transfer-inspired intermediate feature construction + bidirectional InfoNCE + confidence-gated pseudo-labeling, all in single-sample online adaptation.
  • Open-set single-image TTA setting: Explicit formalization and benchmark evaluation of a streaming, single-sample, open-set adaptation protocol.

Results

  • CIFAR-100C→SVHN (CLIP): +8.45% AccHM, −10.80% FPR95, +11.04% AUROC vs. ROSITA (prior best)
  • CIFAR-10C→SVHN (CLIP): +1.58% AUROC, −12.88% FPR95, +1.50% AccHM vs. ROSITA
  • ImageNet-C→MNIST (CLIP): +0.47% AUROC, −7.49% FPR95, +3.75% AccHM vs. ROSITA
  • VisDA→MNIST-M (CLIP): +1.58% AUROC, −7.46% FPR95, +1.09% AccHM vs. ROSITA
  • Varying OOD ratios (0.2–1.0, CIFAR-10C→MNIST): AccHM fluctuates by only 1.01%, outperforming all baselines at every ratio
  • Backbone robustness: Achieves best AccHM and FPR95 across ViT-B/16, ViT-B/32, and ResNet50 on CIFAR-100C→MNIST
  • Ablation: Removing Lmid degrades FPR95 by up to +5.94%; removing intermediate-domain (λ=0) hurts by up to +5.94% FPR95

Limitations

  • The sliding-window feature/score banks (size 128) introduce a dependency on accumulated historical samples, which contradicts the strict single-image premise—performance may degrade at the very start of a stream.
  • Evaluated only on classification benchmarks (digit/CIFAR/ImageNet); authors acknowledge the method has not been tested on detection or segmentation tasks.
  • OOD detection performance saturates less cleanly on complex ID datasets (ImageNet-C), reducing AccHM gains in those settings.
  • No explicit handling of catastrophic forgetting or model drift over extended single-image streams.
  • Requires pre-defined ID class names for prompt construction, limiting applicability to truly unseen semantic categories without any label knowledge.

Relevance to Vision-Language Models

This work directly extends the test-time adaptation paradigm for CLIP and MaPLe to open-set single-sample scenarios, demonstrating that VLMs' cross-modal representations are rich enough to support per-sample ID/OOD classification via dual-polarity prompt engineering. The polarity-aware scoring mechanism is a novel use of negative prompts (popularized in CLIPN/NegPrompt) combined with contrastive self-supervision during inference, advancing VLM-based OOD detection beyond frozen zero-shot evaluation. For researchers tracking VLMs, this paper illustrates a broader trend of exploiting VLM feature geometry at test time for robustness—complementing prompt tuning (TPT, DiffTPT) with feature-space structural constraints. It also highlights a productive tension: VLMs' strong priors enable single-image adaptation, but the method still requires accumulating feature statistics, surfacing a fundamental open question about truly zero-accumulation online VLM adaptation.