Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs

🕒 Published (v1): 2026-07-18 09:24 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Why this paper was selected

MICCAI 2026; test-time modality generalization for medical VLMs; directly buildable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoBE is an optimization-free test-time adaptation framework for medical VLMs that resolves the specialization-generalization dilemma in Mixture-of-Experts settings by combining entropy-guided dynamic expert routing with per-expert Bayesian posterior updates. It requires no gradient computation at inference, yet achieves average accuracy gains of +4.72, +7.17, and +4.3 over prior state-of-the-art TTA methods on seen, unseen, and heterogeneous medical benchmarks respectively. The method is accepted at MICCAI 2026.

Problem

Medical VLMs (MVLMs) suffer accuracy collapse under distribution shift from unseen imaging modalities, scanners, and protocols. Prior MoE-based approaches fall into a specialization-generalization dilemma: uniform aggregation of all modality experts dilutes modality-specific knowledge, while committing to a single expert risks catastrophic mismatch when the test modality is ambiguous or absent from training. Existing gradient-based TTA methods (TENT, TPT, MoME) require backpropagation, while cache-based methods (TDA, BoostAdapter) operate on a single model and do not address expert selection under modality shift.

Method

MoBE augments a frozen MVLM backbone (BiomedCLIP ViT-B/16) with \(E=5\) modality-specialized MLP experts in parallel, pre-trained on disjoint modality partitions of ROCOV2, then performs two-stage optimization-free adaptation at test time:

A. Dynamic-\(k\) Entropy-Guided Routing. For each test sample, predictive entropy \(H(p_e) = -\sum_c p_e(c)\log p_e(c)\) is computed for all \(E\) experts. Experts are sorted by ascending entropy \(H_1 \le \cdots \le H_E\) and the active set is: $\(\mathcal{E}_k = \{e_j \mid H_j - H_1 < \tau\}, \quad k = |\mathcal{E}_k|\)$ with \(\tau\) a modality-agnostic threshold. Selected experts are combined with inverse-entropy weights \(w_e \propto 1/\exp(H(p_e))\), producing gated logits \(s = \sum_{e \in \mathcal{E}_k} w_e s_e\).

B. Expert Bayesian Adaptation (EBA). For each selected expert, two persistent per-class statistics are maintained across the test stream without parameter updates: class prototypes \(\mu_{e,c} \in \mathbb{R}^D\) (initialized from frozen text embeddings) and a class-prior vector \(\Pi_e \in \Delta^{C-1}\) (initialized uniformly). Posterior predictions are: $\(P(y|x) \propto \text{softmax}(\tau_{\text{EBA}} \cdot h^\top \mu_{e,c}) \odot \Pi_e\)$ Confidence-gated running averages update prototypes when \(\max_c P(y|x) > \theta_1\) and update priors at a higher threshold \(\theta_2\), making updates progressively smaller as the running counter grows. Final predictions fuse routed and EBA-adapted logits: \(s_{\text{final}} = (1-\lambda)s + \lambda s_{\text{EBA}}\).

Key Contributions

• Identification and formalization of the specialization-generalization dilemma in test-time modality adaptation for MoE-based MVLMs.
• MoBE: a fully optimization-free (no backpropagation, no source data) TTA framework combining dynamic-\(k\) entropy routing with per-expert online Bayesian adaptation.
• Empirical validation that self-entropy closely approximates supervised cross-entropy as a routing signal (positive Pearson correlation shown).
• Consistent state-of-the-art gains across seen, unseen, and heterogeneous modality benchmarks with multiple backbone configurations.

Results

• Seen modalities (MedMNIST + Med-VTAB): MoBE achieves 43.41% average vs. MoME 38.69%, TDA 36.20%, BiomedCLIP 33.00%; notable +20.52pp on BreastMNIST (73.08% vs. MoME 52.56%).
• Unseen modalities (MedMNIST): MoBE achieves 38.16% average vs. MoME 30.99%, TDA 22.56%, BiomedCLIP 20.49%; +18.95pp on DermaMNIST (66.00% vs. MoME 57.51%).
• Heterogeneous suite (11 datasets, 9 modalities): MoBE achieves 46.49% average vs. TDA 42.19%, BiomedCLIP 39.24%.
• Compute (ChestMNIST, ~22k samples): MoBE runs at 35 images/sec (no backprop), vs. MoME at 28 images/sec (backprop required); BiomedCLIP alone at 144 images/sec.
• PubmedCLIP backbone: MoBE improves from 23.13% to 29.56% average; with BiomedCLIP from 29.23% to 48.74%, confirming backbone transferability.
• Exception: OrganAMNIST and BloodMNIST remain challenging due to weak modality cues and limited expert-bank coverage respectively.

Limitations

• Multiple expert forward passes increase inference latency ~4.5× vs. single-model BiomedCLIP inference (35 vs. 144 images/sec).
• Performance degrades when expert-bank modality coverage is sparse relative to the test distribution (e.g., BloodMNIST microscopy shift not well covered by the five trained expert modalities).
• Entropy-based routing becomes unreliable when test samples lack strong modality cues (e.g., OrganAMNIST).
• MoBE's dynamic-\(k\) threshold \(\tau\) and EBA thresholds \(\theta_1, \theta_2\) are dataset-specific hyperparameters; no source-free automatic tuning is proposed.
• Evaluation uses a fixed \(E=5\) expert set; performance under a different number of experts or broader expert diversity is not studied.

Relevance to Vision-Language Models

MoBE directly addresses a practical failure mode of medical VLMs—modality distribution shift at deployment—without requiring any fine-tuning of the CLIP-style backbone, making it immediately applicable to frozen foundation models like BiomedCLIP. It demonstrates that entropy, a readily computable quantity from VLM softmax outputs, is a reliable unsupervised proxy for cross-entropy routing quality, which is a broadly applicable insight for VLM inference-time adaptation. The Bayesian adaptation of class prototypes initialized from text embeddings shows how the text-vision alignment in VLMs can be exploited as a structured prior for online distribution tracking. For researchers tracking VLMs, this paper advances the frontier of training-free, optimization-free TTA specifically within MoE-augmented VLM architectures, connecting to the broader line of cache-based (TDA, BoostAdapter) and prompt-tuning (CoOp, TPT) TTA work.