Skip to content

Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction

🕒 Published (v1): 2026-07-17 11:08 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; vMF mixture TTA-transduction; rigorous test-time VLM adaptation theory

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MOON addresses the collapse of test-time transductive methods for VLMs under realistic class imbalance by modeling features on the unit hypersphere with a von Mises-Fisher mixture and dynamically adjusting shrinkage strength via zero-shot priors at both instance and class levels. It is training-free, model-agnostic, and requires no task-specific hyperparameter tuning. On ImageNet it improves zero-shot CLIP by 13.2% and outperforms the strongest baseline by 8.8% across 10 imbalanced scenarios.

Problem

Existing transductive and online TTA methods for VLMs assume uniform class marginals at test time. Under realistic deployment—small batches with severe class imbalance, non-i.i.d. streams, or only a few active classes (\(K_\text{eff} \ll K\))—these methods suffer catastrophic performance collapse because majority classes dominate latent statistics, minority class estimates become unreliable, and errors amplify over iterations (negative transfer). Even anchor-based methods like StatA use a static scalar \(\alpha\), treating all instances and classes as equally reliable, which is suboptimal and requires task-specific tuning.

Method

MOON recasts test-time transduction as penalized likelihood estimation (PLE) with a KL-divergence anchor term \(R(M) = \mathrm{KL}(M' \| M)\). The authors prove that the KL-anchored PLE minimizer yields a closed-form adaptive shrinkage update:

\[\nabla A(\eta_k^\star) = \beta_k \hat{\mu}_k + (1-\beta_k)\mu_k', \quad \beta_k = \frac{n_k}{n_k + \alpha}\]

making class statistics a convex combination of empirical estimates and zero-shot text-anchor priors.

Distribution choice: Features lie on \(\mathcal{S}^{d-1}\), so MOON uses a mixture of von Mises-Fisher (vMF) distributions \(\mathcal{V}_k(\mu_k, \kappa_k)\) whose density is \(p(f_i; \mu_k, \kappa_k) \propto C_d(\kappa_k)\exp(\kappa_k \mu_k^\top f_i)\), initialized from zero-shot text embeddings \(\mu_k' = t_k\).

Dynamic shrinkage — instance level: Each sample's contribution is re-weighted by \(\gamma_i = 1 - H(\hat{y}_i)/\log K\) (entropy-based), suppressing high-entropy (unreliable) assignments during parameter estimation.

Dynamic shrinkage — class level: Instead of a fixed \(\alpha\), a per-class anchor weight \(\alpha_k = 1/\lambda_k\) is set inversely proportional to class confidence \(\lambda_k = \sqrt{\frac{1}{N}\sum_i \hat{y}_{i,k} \cdot \max_i \hat{y}_{i,k}}\) (geometric mean of mean and max confidence). This forces absent/outlier classes (\(\alpha_k \to \infty\), \(\beta_k \to 0\)) to remain near their zero-shot anchor, while confident effective classes (\(\alpha_k \to 1\)) follow empirical estimates.

Optimization alternates via Block Successive Minimization (BSUM): a linearized update for assignments \(z_i\) (Eq. 12) and closed-form updates for \(\mu_k\), \(\kappa_k\) (Eq. 13–14). A single pass per iteration avoids inner-loop overhead.

Key Contributions

  • Theoretical unification of transductive VLM methods under a PLE framework, proving KL-anchored PLE inherently yields adaptive shrinkage over the exponential family.
  • Identification of two failure modes: (1) absence of any anchoring mechanism; (2) static, uniform shrinkage strength across instances and classes.
  • MOON: a vMF mixture model with entropy-based instance reweighting (\(\gamma_i\)) and confidence-inversely-proportional class anchor weights (\(\alpha_k\)), both driven purely by zero-shot priors—no labeled data, no task-specific tuning.
  • Training-free, black-box (no access to model internals), and convergence-guaranteed under BSUM.

Results

  • ImageNet, batch adaptation: MOON improves zero-shot CLIP by +13.2% averaged across 10 imbalanced scenarios; outperforms strongest baseline (StatA) by +8.8%.
  • 11 datasets, two realistic settings (batch adaptation at \(B_s \in \{64, 1000\}\); online adaptation with non-i.i.d. Dirichlet streams): MOON achieves best average accuracy in all scenarios evaluated in Fig. 1 and Tables 1–2.
  • Efficiency: Processing thousands of samples in tens of milliseconds, only 3.3% of CLIP's inference latency.
  • Most competing methods (TransCLIP, GDA-CLIP, ADAPT, ZLaP, OGA, EM-Dirichlet) collapse or degrade significantly under realistic imbalance; StatA partially recovers but remains inferior to MOON.
  • Gains are most pronounced at severe imbalance (\(K_\text{eff}/\min(N,K) \to 0\)) and on large-scale datasets (ImageNet).

Limitations

  • Slight performance drop observed under Online adaptation at Low correlation strength (noted but not fully quantified in the provided text).
  • \(\lambda_k\) uses geometric mean of average and max confidence as a heuristic to balance rare-but-present classes against true outliers; this mildly sacrifices accuracy under very severe imbalance for broader generality.
  • Evaluation backbone is fixed to CLIP ViT-B/16; generalization across architectures (e.g., SigLIP, ALIGN) is asserted as model-agnostic but not exhaustively validated in the provided text.
  • Requires \(\ell_2\)-normalized embeddings on the unit hypersphere; applicability to VLMs without cosine similarity heads is not discussed.

Relevance to Vision-Language Models

MOON directly targets the test-time inference pipeline of CLIP-style VLMs, proposing a plug-in that replaces the zero-shot softmax with a transductive clustering step—without touching model weights. For VLM researchers, it surfaces a fundamental but underexamined failure mode: contrastive VLMs produce embeddings on \(\mathcal{S}^{d-1}\), making the vMF distribution the geometrically correct generative model, yet prior transductive methods defaulted to Gaussian assumptions. The dynamic shrinkage framework provides a principled connection between zero-shot confidence (from text encoders) and adaptation trust, directly operationalizing how VLM text priors should regularize feature-space clustering. This work extends a growing line—TransCLIP, StatA, ZLaP—and demonstrates that the class-imbalance brittleness problem is solvable without training, broadening practical zero-shot deployment of VLMs.