Skip to content

Noisy Test-Time Adaptation in Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper identifies that existing test-time adaptation (TTA) methods fail severely when noisy out-of-distribution samples are present in the test stream, often underperforming a frozen CLIP baseline. The authors formalize this as Zero-Shot Noisy TTA (ZS-NTTA) and propose AdaND, which decouples noise detection from classification by training a lightweight linear detector on frozen VLM features while keeping the backbone and classifier completely frozen. AdaND improves harmonic mean accuracy by 8.32% on ImageNet over prior TTA methods and also advances zero-shot OOD detection by 9.40% in FPR95.

Problem

Standard TTA methods assume all test samples belong to the in-distribution (ID) label space. In open-world deployment, test streams contain noisy (OOD) samples. Existing TTA methods—even when augmented with MCM-score filtering—degrade below the frozen ZS-CLIP baseline because: (1) unfiltered noisy samples produce gradients an order of magnitude larger than clean samples, causing rapid overfitting to noise; (2) jointly adapting the classifier for both ID classification and noise detection corrupts both sub-tasks simultaneously. Prior noisy-TTA works require task-specific source prototypes unavailable for zero-shot VLMs, or ignore noisy-sample recognition entirely.

Method

Framework (decouple classifier from detector): The CLIP backbone and text-based classifier are kept completely frozen throughout test time. Only a single linear noise detector head is trained online.

Adaptive Noise Detector (AdaND): - Uses frozen CLIP's MCM scores with an adaptive Otsu-style threshold (minimizing intra-class variance over a running queue) to generate pseudo-labels for clean vs. noisy samples. - Trains the linear detector via cross-entropy on frozen image features I(x) with these pseudo-labels. - A two-stage detection protocol: for the first N=10 steps, ZS-CLIP detection is used directly; thereafter, the trained detector takes over. - A queue of length L=128 accumulates features/pseudo-labels; update occurs every L samples (handles batch size = 1).

Gaussian Noise Injection (clean-stream robustness): Without any OOD samples, the detector collapses—it assigns everything as noisy. To prevent this, a Gaussian noise image is inserted every M=8 samples regardless of stream type. These synthetic OOD samples anchor the detector's negative class, preventing false-positive noisy classifications on purely clean streams.

ZS-OOD extension: The MCM score from the trained detector's output logits is repurposed as an OOD score for standard zero-shot OOD detection.

Key Contributions

  • Formalization of the ZS-NTTA setting and a benchmark spanning 11 ID datasets × multiple OOD datasets (44 ID-OOD pairs total).
  • Empirical analysis showing that noisy-sample gradients dominate updates by ~10× over clean samples, explaining why adapted models regress below the frozen baseline.
  • AdaND: a classifier-detector decoupling framework with a trainable single linear layer and Gaussian noise injection that is noise-type-agnostic and noise-ratio-agnostic.
  • Demonstration that AdaND transfers to the ZS-OOD detection task without architectural change.
  • Computational parity with frozen ZS-CLIP: 0.1272 s/sample vs. 0.1125 s/sample at bs=1; 3.83 GiB vs. 3.80 GiB GPU memory.

Results

  • ZS-NTTA on ImageNet: AdaND AccH = 73.09% vs. ZS-CLIP 64.77% (+8.32%), vs. best prior TTA (SoTTA) 59.47%. Consistent across ImageNet-A, -V2, -R, -K variants.
  • ZS-NTTA on fine-grained datasets: AdaND achieves average AccH of 66.14% (CUB-200), 76.91% (Stanford-Cars), 92.23% (Food-101), 91.54% (Oxford-IIIT Pet), outperforming all baselines across all four OOD pairings per dataset.
  • ZS-OOD detection on ImageNet: AdaND avg AUROC = 95.58% (+1.37% vs. NegLabel), avg FPR95 = 16.00% (−9.40% vs. NegLabel 25.40%). Achieved without auxiliary corpus or extra text encoder training.
  • Ablation: Removing the noise detector drops noisy-stream AccN from 88.67% to 82.38%; removing Gaussian noise collapses AccS on clean streams from 63.96% to 37.54%.
  • Runtime (ImageNet, bs=1): ZS-CLIP 0.1125 s / 3.80 GiB; AdaND 0.1272 s / 3.83 GiB; TPT 0.3219 s / 21.23 GiB.

Limitations

  • Evaluation is confined to CLIP (ViT-B/16) as the VLM backbone; generalization to other VLM families (e.g., LLaVA, Flamingo) is not shown in the main paper.
  • The adaptive threshold and pseudo-label quality rely on ZS-CLIP being reasonably accurate; performance may degrade if the base VLM is weak on the target ID classes.
  • Gaussian noise injection frequency M and queue length L are tuned to fixed values that may not be optimal for all noise ratios or domain shifts.
  • The method addresses sample-level OOD noise but does not handle label-level corruption or semantic shifts within the ID label space.
  • Experiments with varying noise ratios and random seeds are limited to CIFAR-10/100 due to compute constraints.

Relevance to Vision-Language Models

This paper directly targets a practical failure mode of CLIP-based zero-shot classification: the open-world assumption that test streams are ID-only. For researchers tracking VLMs, it demonstrates that zero-shot CLIP's frozen representations are surprisingly robust as a classifier, but that adaptation strategies must treat the backbone as inviolable when OOD contamination is present. The decoupled detector paradigm—training only a lightweight head on frozen features—is a broadly applicable principle for any VLM deployment requiring robustness without forgetting. The secondary result on ZS-OOD detection also shows that test-time learning of a discriminative feature boundary can improve upon purely score-based OOD methods, offering a new angle for VLM uncertainty quantification research.