BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming¶

🕒 Published (v1): 2026-06-23 16:03 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; confusion-aware mixture-of-prompt experts for biomedical CLIP adaptation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

BioMedVR is the first Visual Reprogramming (VR) framework applied to biomedical imaging, adapting frozen CLIP models via dual learnable input-space perturbations guided by LLM-generated confusion-aware attributes. It introduces a Mixture-of-Prompt Experts (MoPE) with a positive expert for main-class alignment and a negative expert for confusion suppression, combined via adaptive gating. On 11 biomedical datasets, BioMedVR achieves 76.4% average accuracy in 16-shot settings using only 0.30M trainable parameters.

Problem¶

CLIP and similar VLMs generalize well to natural images but fail on biomedical imaging due to high inter-class similarity (e.g., cataract vs. glaucoma) and scarce labeled data. Existing VR methods apply a single shared visual prompt across all classes, which (i) provides weak discriminative signal when class text embeddings are nearly identical (\(f_t(t_{y_i})^\top f_t(t_{y_j}) \approx 1\)), and (ii) cannot disentangle the heterogeneous Jacobian structure across imaging modalities (MRI, CT, ultrasound), leading to overconfident, miscalibrated predictions. Prompt-learning methods (CoOp, MaPLe) require access to model internals and still ignore confusing negatives.

Method¶

BioMedVR learns two input-level perturbations \(\delta^+\) and \(\delta^-\) applied additively to each image:

\[\tilde{x}^+ = x + \delta^+, \quad \tilde{x}^- = x + \delta^-\]

Positive expert \(\delta^+\): guided by descriptive, discriminative attributes \(\mathcal{T}^+_y\) (following AttrVR), computes top-\(k\) averaged alignment scores \(s^+_y\).

Negative expert \(\delta^-\): guided by confusion-aware attributes \(\mathcal{T}^-_y\) generated by querying an LLM (GPT-4.1 by default) with "Generate the most visually confusing negative descriptions for class [y]", yielding \(s^-_y\).

Expert outputs are fused via a learnable 2D gating vector \(w\):

\[s_y = g^+ s^+_y + g^- s^-_y, \quad [g^+, g^-] = \text{softmax}(w)\]

The Confusion-Suppression (CS) loss enforces a semantic margin \(\Gamma(x,y) = s^+_y - \max_{c \neq y} s^-_c \geq m\):

\[\mathcal{L}_{cs} = \mathbb{E}_{(x,y)}\left[\max\!\left(0,\, \max_{c \neq y} s^-_c - s^+_y + m\right)\right]\]

Final objective: \(\mathcal{L} = \mathcal{L}_{CE} + \beta\,\mathcal{L}_{cs}\) with \(\beta=0.3\), \(m=0.5\). All CLIP encoder weights remain frozen; only \(\{\delta^+, \delta^-, w\}\) are trained (0.30M parameters total).

Key Contributions¶

First application of Visual Reprogramming to biomedical imaging, enabling architecture-agnostic, privacy-preserving few-shot VLM adaptation without touching model internals.
Confusion-aware MoPE: dual-expert design explicitly disentangling discriminative (positive) and confusable (negative) semantics via LLM-generated confusion attributes.
Confusion-Suppression Loss: margin-based hinge that penalizes insufficient separation between the true class score and the highest-scoring confusing class, activating only when confusion occurs.
Adaptive gating module that automatically balances the two experts during training without manual weighting.
Validated across 11 biomedical datasets (9 imaging modalities) + 7 natural image benchmarks, covering both few-shot and zero-shot settings.

Results¶

16-shot average accuracy across 11 biomedical datasets: BioMedVR 76.4% vs. AttrVR 73.0% vs. AR 72.6% vs. BiomedCoOp 71.2% vs. CoCoOp 66.6%.
Notable dataset gains over AttrVR: Knee X-ray (45.7% vs. 33.5%), BUSI (82.6% vs. 78.0%), COVID-19 (77.4% vs. 71.0%).
Zero-shot with BioMedCLIP backbone: 47.6% avg vs. BioMedCLIP zero-shot 40.9%.
Natural image benchmarks (16-shot): BioMedVR matches or exceeds AttrVR on all 7 datasets (e.g., DTD 66.0% vs. 65.9%, EuroSAT 93.9% vs. 93.4%, Flowers 93.0% vs. 93.1%).
Parameter efficiency: 0.30M trainable params vs. ~150M for full CLIP fine-tuning (>500× reduction); training time 9.10 s/epoch vs. 6.55 s for AttrVR.
Sample efficiency: BioMedVR outperforms AttrVR and BiomedCoOp at 4-shot, 8-shot, and 16-shot on representative biomedical datasets.

Limitations¶

Adds a second learnable perturbation \(\delta^-\), doubling parameter count relative to single-expert VR methods (0.30M vs. 0.15M), with ~40% longer training time per epoch.
Quality of confusion-aware attributes depends on the LLM used; the paper evaluates GPT-4.1, GPT-4o-mini, GPT-5-mini, and GPT-5 but does not report results with open-source or locally deployable LLMs.
Zero-shot BioMedVR (without BioMedCLIP) underperforms BioMedCLIP zero-shot on several datasets, indicating that a general-domain CLIP backbone still limits zero-shot biomedical coverage.
Margin hyperparameter \(m\) and loss weight \(\beta\) require dataset-level tuning via hyperparameter search.
Evaluated only on classification tasks; generalization to detection, segmentation, or report generation is not demonstrated.

Relevance to Vision-Language Models¶

BioMedVR directly addresses the domain-adaptation bottleneck for CLIP-family VLMs in high-stakes specialized domains, demonstrating that input-space reprogramming can outperform both prompt-tuning and adapter-based PEFT methods without modifying model internals—a significant practical advantage for privacy-sensitive medical deployments. The confusion-aware MoPE design highlights a broadly applicable principle: explicitly modeling hard negatives in the textual attribute space (via LLM generation) improves VLM calibration in fine-grained settings where standard contrastive objectives produce flat softmax distributions. This connects directly to ongoing work on negative-aware prompting, attribute-guided adaptation, and few-shot medical VLMs (BiomedCoOp, AttrVR, MaPLe), and suggests that LLM-generated adversarial text can serve as a cheap supervision signal to sharpen VLM decision boundaries across any fine-grained domain.