Skip to content

DOTA: Distributional Test-time Adaptation of Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DOTA replaces the per-sample cache paradigm in test-time adaptation (TTA) of VLMs with online Gaussian distribution estimation per class, using zero-shot predictive probabilities as soft EM weights. This eliminates cache-size-induced catastrophic forgetting and enables continuous, gradient-free adaptation to the test stream.

Problem

Cache-based TTA methods (e.g., TDA, BoostAdapter) are constrained by finite cache capacity: they store only a limited set of representative samples, discard older entries, and consequently suffer catastrophic forgetting as the test distribution evolves. They also fail to capture the full statistical structure of the test stream, limiting both accuracy and long-horizon adaptability.

Method

DOTA assumes per-class image embeddings follow a Gaussian distribution \(P(x|y=k) = \mathcal{N}(\mu_k, \Sigma_k)\) and maintains running estimates \((\hat{\mu}_k, \hat{\Sigma}_k)\) for each class \(k\). At each test step, it: 1. Soft-labels incoming samples using CLIP's zero-shot softmax probabilities \(P_k^{zs}\) as EM weights (avoiding the need for ground-truth labels). 2. Updates \(\hat{\mu}_k\) and \(\hat{\Sigma}_k\) via a streaming online algorithm (Dasgupta & Hsu 2007) that accumulates confidence-weighted sufficient statistics, converging with zero average regret. 3. Classifies using the Bayesian posterior \(P(y=k|x) \propto \exp(f_k(x))\) where \(f_k(x) = -\frac{1}{2}(x-\hat{\mu}_k)^T \hat{\Sigma}_k^{-1}(x-\hat{\mu}_k) - \frac{1}{2}\log|\hat{\Sigma}_k|\). 4. Fuses zero-shot and distributional logits as \(\cos(x,w_k)/\tau + \lambda f_k(x)\), where \(\lambda = \min(\rho c, \eta)\) ramps up with sample count \(c\), suppressing the unreliable distributional term early in the stream.

To reduce inversion cost from \(K\) to 1, covariances are averaged across classes; shrinkage regularization \(\hat{\Lambda} = [(1-\epsilon)\hat{\Sigma} + \epsilon I]^{-1}\) (\(\epsilon=10^{-4}\)) ensures positive definiteness. No gradient backpropagation is required.

Key Contributions

  • A distributional TTA paradigm for VLMs that estimates per-class Gaussians online rather than caching individual samples, eliminating the capacity-forgetting trade-off.
  • A streaming EM update rule using zero-shot probabilities as soft labels, enabling label-free, one-pass distribution estimation from non-stationary test streams.
  • A sample-count-adaptive logit fusion that interpolates zero-shot and distributional classifiers, protecting against unreliable early-stage estimates.
  • Validation on 10 cross-domain datasets, 5 ImageNet-shift datasets, and 3 pathology datasets with an alternative foundation model (PLIP), plus inference-time benchmarks showing >20× speedup over prompt-based TTA.

Results

  • Cross-domain generalization (10 datasets, ViT-B/16): DOTA average 69.71% vs. BoostAdapter 68.68% (+1.03%), TDA 67.53%, TPT 65.10%, zero-shot 64.59%.
  • Natural distribution shifts (ImageNet variants, ViT-B/16): DOTA average 66.31% vs. TDA 65.10% (+1.21%), ZERO 63.64%, TPT 62.19%.
  • Pathology (PLIP model): DOTA average 70.56% vs. TDA 63.67% (+6.89%), zero-shot 61.80%.
  • Inference speed (ImageNet, ViT-B/16): DOTA 22 min (same as TDA), vs. TPT 447 min (24×), DiffTPT 1346 min (61×).
  • Continuous learning (last 50% of stream): DOTA consistently improves on the latter half; TDA degrades on several datasets, confirming forgetting in cache-based methods.
  • Ablation: Using high-confidence pseudo-labels only (vs. soft EM) costs −2.49% avg; removing covariance updates costs −2.39% avg, with −7.72% on EuroSAT.

Limitations

  • Single Gaussian per class cannot capture multimodal or non-Gaussian intra-class distributions.
  • Uniform class prior (\(P(y=k)=1/K\)) is incorrect under real-world class imbalance; no adaptive prior mechanism.
  • Logit fusion weight schedule (\(\lambda = \min(\rho c, \eta)\)) is heuristic; no principled uncertainty-aware weighting.
  • Performance degrades below TDA when per-class sample count is very low (e.g., ImageNet-V2 with only 10 samples/class), since Gaussian estimates are unreliable in this regime.

Relevance to Vision-Language Models

DOTA directly targets the deployment reliability of CLIP-class VLMs under distribution shift — a known Achilles heel of large-scale pretrained models — without touching model weights or prompts. Its gradient-free, streaming design is practically significant for on-device and latency-sensitive VLM deployment where backpropagation is infeasible. The EM-with-soft-labels formulation generalizes naturally to any VLM producing calibrated softmax outputs, demonstrated here on both CLIP and a domain-specific pathology VLM (PLIP). For the broader TTA-for-VLMs literature, DOTA represents a principled move from instance-level to distributional memory, a direction orthogonal to prompt tuning and adapter methods.