Statistics Caching Test-Time Adaptation for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SCA (Statistics Caching test-time Adaptation) is a training-free, cache-based test-time adaptation method for CLIP-style VLMs that replaces raw feature storage with incrementally updated feature statistics (Gram matrix + weighted feature-label matrix), eliminating forgetting and cache-blocking issues. It adds a dynamic soft pseudo-labeling strategy and an instance-level entropy-based fusion mechanism. On 15 datasets it matches or surpasses SOTA cache-based and prompt-tuning TTA methods at over 10× the speed of TPT.

Problem

Existing TTA methods for VLMs suffer from one of two failure modes: (1) test-time prompt tuning methods reset per sample and cannot accumulate cross-sample knowledge, and (2) cache-based methods store a fixed number of raw features per class, making them vulnerable to forgetting (a confident misclassified sample overwrites a correct entry) and blocking (entrenched high-confidence wrong samples prevent better features from entering). Neither achieves robust, continuous knowledge accumulation from all test samples.

Method

SCA recasts knowledge accumulation as a ridge regression problem over all past test features. Rather than storing the growing feature matrix X_{1:t}, it maintains two compact, size-invariant statistics: the Gram matrix G_{1:t} = X^⊤X (d×d) and the feature-label co-statistic C_{1:t} = X^⊤Y (d×K). Both are updated additively at each step (O(d²) per sample), and the closed-form classifier W_t = (G_{1:t} + γI)⁻¹ C_{1:t} is computed from these alone. Because every sample contributes additively, no sample can be "evicted" and prior knowledge is never overwritten.

Pseudo-labels for updating C_{1:t} use a dynamic soft assignment: a cumulative-probability threshold τ identifies a candidate class set Ω per sample, then a softmax over the logits of those classes yields a soft label—sharper when the model is confident, softer when uncertain—replacing hard argmax pseudo-labeling.

Final predictions combine the statistics-derived cache logit and the CLIP zero-shot textual logit via instance-level adaptive fusion: entropy of each classifier's prediction determines per-sample weights via softmax(-β[H_cache, H_text]), so the cache is trusted more as it becomes confident, with a single β = 0.5 working across all datasets.

Key Contributions

• Statistics-based cache: replaces raw feature storage with the Gram matrix G and co-statistic C, enabling lossless accumulation from all test samples without forgetting or blocking.
• Dynamic soft pseudo-labeling: per-sample candidate class set derived from cumulative probability threshold; produces soft labels that track sample-level uncertainty rather than collapsing to hard one-hot assignments.
• Instance-level entropy-based adaptive fusion: automatically weights textual vs. cache logits per sample using prediction entropy; eliminates the need for per-dataset hyperparameter search on fusion coefficients.
• Storage scales as (K + d) × d rather than M × K × d, becoming more favorable than fixed-size caches as the number of classes K grows.

Results

Cross-domain benchmark (10 datasets): - ViT-B/16: SCA 70.34% avg vs. DPE 69.40% (+0.94%), TDA 67.53% (+2.81%), BCA 68.59% (+1.75%), Dynaprompt 65.52% (+4.82%) - ResNet-50: SCA 63.29% avg vs. DPE 61.93% (+1.36%), TDA 61.03% (+2.25%)

OOD benchmark (ImageNet + 4 variants): - ViT-B/16: SCA 66.16% avg (64.77% OOD-only avg) vs. DPE 65.93%/64.43%, BCA 65.37%/64.16% - ResNet-50: SCA 50.49% avg, slightly below DPE 50.81% on this backbone

Computation (SUN397, ViT-B/16, single A100): - SCA: 0.012 s/image; TPT: 0.218 s/image (>10× faster); TDA: 0.009 s/image; DPE: 0.043 s/image

Ablation (ViT-B/16, cross-domain avg): - Full SCA: 70.34%; w/o statistics accumulation (fixed cache M=32): 69.72%; w/o dynamic soft labels: 69.32%

Limitations

• Storing the d×d Gram matrix introduces quadratic-in-d storage overhead, worse than fixed caches when K is small.
• Soft pseudo-labels mitigate but do not eliminate error accumulation; performance still falls short of an oracle classifier built from ground-truth labels.
• Evaluated in single-domain TTA only; naive statistics accumulation across mixed-domain test streams could be harmful due to distributional mismatch between domains.
• The closed-form update requires a matrix inversion (G_{1:t} + γI)⁻¹ C_{1:t} at each step, which may become a bottleneck for very large feature dimensions d.

Relevance to Vision-Language Models

SCA addresses a core bottleneck in deploying CLIP-style VLMs without labeled target data: test-time distribution shift. By replacing raw-feature caching with second-order sufficient statistics, it demonstrates that the standard cache paradigm (store-select-replace) is fundamentally lossy and that a closed-form least-squares alternative achieves strictly better knowledge retention. The soft pseudo-label and entropy-based fusion strategies are broadly applicable to any cache-augmented VLM inference pipeline. This work extends a growing body of TTA research (TPT, TDA, DPE) toward more principled statistical estimators rather than heuristic cache management, pointing toward future work on multi-domain and continual adaptation of large vision-language models.