Skip to content

Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Mint is a test-time adaptation (TTA) method for CLIP that addresses performance degradation under image corruptions by identifying and counteracting embedding variance collapse — a phenomenon where both intra-class and inter-class embedding variances shrink as corruption severity increases. It maximizes pseudo-label-based inter-class variance using cumulative mean and gradient accumulators, enabling effective adaptation even at batch size 1.

Problem

CLIP's zero-shot performance degrades under common image corruptions (noise, blur, weather, digital), but existing TTA methods either focus on text-side adaptation (prompt tuning, ensemble) or require large batches for stable normalization-layer updates. No prior work theoretically explains why corruptions degrade CLIP embeddings or provides a principled, small-batch-compatible fix.

Method

Mint adapts only the LayerNorm parameters of CLIP's visual encoder by maximizing pseudo-label inter-class variance (PL-inter) online:

  1. Variance decomposition: PL-inter is rewritten as PL-total minus PL-intra, enabling stable gradient computation without full-dataset access.
  2. Mean accumulator: Maintains running cumulative averages of per-class and global image embeddings (updated online via Eq. 7), replacing noisy single-batch mean estimates.
  3. Gradient accumulator: Keeps a running average of per-batch gradients (Eq. 9), reducing noise in parameter updates — effectively simulating a large effective batch.
  4. Text embedding adjustment: Interpolates original text embeddings with accumulated per-class image means (Eq. 10), with prior strength K_prior controlling the blend ratio.
  5. After each batch, model weights and optimizer state are reset to initialization; only the accumulators persist across batches. One gradient-ascent step is taken per batch.

Theoretical results (Theorems 3.1, 3.2) prove that corruption severity causes variance collapse via structured shift features diluting class-discriminative dimensions, and that PL-inter maximization provably suppresses corruption-related LayerNorm weights while amplifying class-relevant ones.

Key Contributions

  • Empirical identification and theoretical characterization of embedding variance collapse in CLIP under corruptions, showing GT-inter variance correlates with accuracy at r = 0.98.
  • Theoretical proof (Theorem 3.2) that PL-inter maximization via LayerNorm updates is a valid proxy for GT-inter maximization under noisy pseudo-labels.
  • Mint algorithm: online, single-update-per-batch TTA that works at batch size 1 via dual accumulators.
  • Consistent SOTA across CIFAR-10-C (ViT-B/32), CIFAR-100-C (ViT-B/16), and ImageNet-C (ViT-L/14) with strong computational efficiency.

Results

  • CIFAR-10-C (ViT-B/32): Mint 71.0% avg vs. best baseline WATT-S 67.1% (+3.9 pp).
  • CIFAR-100-C (ViT-B/16): Mint 44.1% vs. WATT-S 41.9% (+2.2 pp).
  • ImageNet-C (ViT-L/14): Mint 47.0% vs. WATT-S 43.9% (+3.1 pp).
  • Batch size robustness: Mint at BS=1 achieves 70.5/43.1/45.8% on the three benchmarks, negligibly below BS=20 (71.0/44.1/47.0%).
  • Efficiency: Mint takes 1m07s per corruption type on CIFAR-100-C vs. 50m20s for WATT-S and 7m40s for CLIPArTT, while outperforming both.
  • Both accumulators are individually necessary; ablation shows removing either degrades performance, especially at small batch sizes.

Limitations

  • Theorems assume a simplified RMSNorm (no demeaning/bias) and an infinite-sample limit; finite-sample behavior is empirically validated but not fully characterized.
  • Adaptation resets model weights each batch, which means no persistent parameter improvement — only the accumulators carry state; this design choice trades fine-grained adaptation for stability.
  • Pseudo-label quality is assumed sufficient (Theorem 3.2 requires pseudo-labels to correlate with ground truth above a noise threshold); severely corrupted inputs may violate this.
  • Hyperparameter K_prior and learning rate are shared across architectures and datasets via a coarse grid — no formal selection criterion is provided.
  • Evaluated only on CLIP; generalization to other VLM architectures (e.g., SigLIP, BLIP-2) is not demonstrated.

Relevance to Vision-Language Models

Mint directly targets a fundamental robustness failure mode in CLIP — the collapse of discriminative structure in image embeddings under distribution shift — and provides the first theoretical account of why this occurs in terms of the encoder's geometry. For VLM researchers, this reframes corruption robustness as an embedding-quality problem rather than a text-image alignment problem, suggesting that the image encoder's normalization layers are the critical adaptation target. The work complements prompt-tuning and ensemble TTA approaches by showing that image-side adaptation is orthogonal and highly complementary, and the accumulator design offers a blueprint for stable online adaptation in streaming deployment scenarios where batch size is severely constrained.