Skip to content

TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models

🕒 Published (v1): 2026-04-17 06:59 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Test-time textual learning improves CLIP-based OOD detection without training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TTL introduces a test-time textual learning framework for out-of-distribution detection that dynamically learns OOD-discriminative text prompts from unlabeled test streams, eliminating dependence on fixed external OOD labels. Rather than adapting visual features against a frozen text space, TTL optimizes learnable OOD prompts per ID class via pseudo-labeled test samples, a noise-purification loss, and an accumulated OOD Textual Knowledge Bank for stable score calibration.

Problem

CLIP-based test-time adaptation methods for OOD detection (e.g., AdaNeg) rely on a finite, pre-specified set of external OOD text labels to represent the OOD semantic space. Because real OOD distributions are open-ended and constantly evolving, these fixed labels fail to cover novel OOD semantics encountered in test streams, leaving a systematic coverage gap that degrades detection on out-of-scope OOD inputs.

Method

TTL operates entirely in the text modality during test time using CLIP (ViT-B/16) with frozen image and text encoders.

Learnable OOD Prompts. For each of the \(N\) ID classes, TTL maintains a learnable OOD prompt \(u_i^{ood}\) initialized to the manual template "a photo of a [classname]." Only the prefix tokens are updated; class name tokens and all encoders are frozen. The OOD probability for a test sample \(x\) is: $\(p(x) = \frac{\sum_{k=1}^{N} s(x, t_k^{ood})}{\sum_{j=1}^{N} s(x, t_j^{id}) + \sum_{j=1}^{N} s(x, t_j^{ood})}\)$ where \(s(x,t)=\exp(\cos(f(x),t)/\tau)\).

OOD-Focused Minority-Balanced Loss (\(\mathcal{L}_{OMB}\)). Prompts are optimized using pseudo-labels from a base detector (MCM), with class-proportion reweighting \(\pi^+\)/\(\pi^-\) to handle imbalanced ID/OOD ratios: $\(\mathcal{L}_{OMB} = -\frac{1}{\pi^+}\sum_{i:\hat{y}_i=1}\log(1-p(x_i)) - \frac{1}{\pi^-}\sum_{j:\hat{y}_j=0}\log p(x_j)\)$

OOD Knowledge Purification (\(\mathcal{L}_{OKP}\)). Within each batch's pseudo-OOD set, an adaptive threshold \(\theta\) (derived via intra-class variance minimization on OOD probabilities) splits samples into high-confidence \(S_h\) and low-confidence \(S_\ell\) (ID boundary noise). The purification loss increases similarity for \(S_h\) and suppresses it for \(S_\ell\): $\(\mathcal{L}_{OKP} = -\left(\frac{1}{|S_h|}\sum_{i\in S_h}p(x_i) - \frac{1}{|S_\ell|}\sum_{j\in S_\ell}p(x_j)\right)\)$ Final loss: \(\mathcal{L} = \mathcal{L}_{OMB} + \alpha \cdot \mathcal{L}_{OKP}\).

OOD Textual Knowledge Bank (OKB). A fixed-capacity bank (\(K=2048\)) accumulates the most discriminative OOD text features across batches, scored by minimum negative cosine similarity to all ID text features. At inference, the calibrated score uses the bank: \(S_{cal}(x) = -\max_{j\in\{1,\ldots,K\}}\cos(f(x), t_j^{ood})\), fused as \(S_{final}(x) = S_{base}(x) + \beta \cdot S_{cal}(x)\).

Key Contributions

  • Test-time textual learning framework (TTL) that adapts OOD text representations from unlabeled test streams without any external OOD labels.
  • OOD Knowledge Purification strategy (\(\mathcal{L}_{OKP}\)) using adaptive intra-class variance thresholding to separate reliable OOD pseudo-labels from ID boundary noise.
  • OOD Textual Knowledge Bank (OKB) with discriminativeness-based retention that accumulates cross-batch textual knowledge for stable score calibration.
  • First application of prompt learning to the test-time OOD detection setting.

Results

  • ImageNet-1k benchmark (4 OOD datasets: iNaturalist, SUN, Places, Texture): TTL achieves 12.46% FPR95 / 97.29% AUROC on average, vs. AdaNeg (19.22% / 96.17%) and OODD (23.64% / 94.09%) among TTA methods; outperforms best training-based method MoFE by 7.56% FPR95.
  • CIFAR-100 benchmark (6 OOD datasets): TTL achieves 2.36% FPR95 / 99.26% AUROC on average, vs. AdaND (20.95% / 92.50%) and AdaNeg (40.52% / 88.97%); on Places365, TTL delivers +19.9% AUROC improvement over other TTA methods.
  • Ablation: each of \(\mathcal{L}_{OMB}\), \(\mathcal{L}_{OKP}\), and OKB contributes incrementally; adding \(\mathcal{L}_{OKP}\) yields +1.03% average AUROC across benchmarks.
  • TTL improves over all tested base detectors (MCM, GL-MCM, Neglabel, LoCoOp, FA); combined with FA: 5.88% FPR95 / 98.76% AUROC on ImageNet-1k.
  • Combined gains over prior best TTA method (AdaNeg): 12.67% FPR95 and 3.94% AUROC average improvement.

Limitations

  • The pseudo-label quality of the base OOD detector constrains the upper bound of TTL's learned OOD knowledge; degraded base detectors propagate noise into prompt optimization.
  • The OKB retention strategy and \(\beta\) fusion coefficient are tuned per benchmark (different \(\beta\) for ImageNet-1k vs. CIFAR-100), reducing zero-shot deployability.
  • Evaluation is restricted to two benchmark families (ImageNet-scale and CIFAR-scale); generalization to domain-shift-heavy settings (medical imaging, autonomous driving) is not demonstrated.
  • The open-ended OOD semantic space assumption means TTL still cannot guarantee coverage of entirely novel visual concepts not encountered in the test stream.

Relevance to Vision-Language Models

TTL directly exploits the dual-encoder structure of CLIP—keeping the text modality learnable at test time while freezing vision—demonstrating that the textual branch of VLMs is an underutilized axis for post-deployment adaptation. This work extends the growing body of prompt tuning research (CoOp, ProDA) into the unsupervised, streaming test-time regime, showing that CLIP's joint embedding space is rich enough to encode emergent OOD semantics without any labeled supervision. For researchers tracking VLMs, TTL provides a concrete architecture pattern—learnable OOD prompts + knowledge bank + noise purification—that generalizes beyond OOD detection to any test-time distribution shift problem where the text modality can absorb new semantic knowledge. It also surfaces a limitation of fixed-text-space VLM adaptation that is broadly relevant to retrieval, open-vocabulary detection, and zero-shot classification pipelines.