Skip to content

Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging

🕒 Published (v1): 2025-05-27 19:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedBridge is a lightweight adaptation framework that repurposes frozen general-purpose vision-language models (VLMs) for multi-label medical image diagnosis without retraining from scratch. It jointly addresses domain shift, high-resolution requirements, and multi-label heterogeneity through learnable query injection and a query-driven mixture-of-experts (MoE) routing over heterogeneous VLMs. On five chest X-ray benchmarks, it achieves 6–15% AUC improvement over state-of-the-art adaptation methods.

Problem

Direct transfer of foundation VLMs (e.g., CLIP) to medical imaging fails due to three simultaneous constraints not addressed jointly by existing methods: (1) medical images are high-resolution (~2048×2048) with spatially sparse pathological cues that vanish under standard 224×224 downsampling; (2) severe domain shift between natural and medical image statistics risks destroying pretrained knowledge if weights are perturbed; (3) clinical diagnosis is intrinsically multi-label with disease-specific cues requiring multi-scale reasoning (e.g., cardiomegaly vs. pneumothorax demand global vs. local features). Training dedicated medical foundation models from scratch is prohibitively costly.

Method

MedBridge wraps each frozen VLM in a Multi-view Query Encoder (MQEncoder) that extracts N=11 local 512×512 crops via a sliding-window focal sampling strategy, resizes to 224×224, and passes all views through the frozen ViT backbone. Starting at intermediate layer n₁+1, a compact set Q of learnable query tokens (Q∈[24,36], ~4% parameter overhead) are concatenated with frozen tokens and processed jointly through remaining layers via standard MHSA+FFN. After the final layer, query tokens are split out and fused with frozen tokens via p_k = m̄_k + α·q̄_k.

A query-driven MoE takes the multi-view query tokens from K heterogeneous VLM experts as input to a softmax gating network (no shared representation space required), producing a weighted sum over expert prediction tokens. Text labels ("a radiology image with [CLASS]") are encoded by each model's frozen text encoder, then a learnable projection aligns them to the vision embedding space. Multi-label binary cross-entropy on cosine similarities trains only the query tokens, gating parameters, and projection layer; all backbone weights remain frozen.

Key Contributions

  • MQEncoder: learnable query injection into frozen intermediate ViT layers for non-destructive domain alignment, combined with multi-view focal sampling to preserve high-resolution pathological detail.
  • Query-driven MoE routing: queries double as routing signals, enabling architecture-agnostic integration of heterogeneous VLMs (CLIP, SigLIP, DINOv2, LLaVA, Qwen3-VL, MedGemma) without a shared embedding space.
  • Model-agnostic framework: validated across eight VLMs spanning general-purpose (CLIP, LLaVA, Qwen3-VL) and medical-specific (CheXzero, MedGemma) backbones.
  • Three adaptation regimes: cross-domain (general→medical), in-domain (medical→new disease), and zero-shot cross-dataset transfer, all evaluated under varying label budgets (1%–100%).

Results

  • Cross-domain AUC (Table 1, CLIP backbone vs. MMRL best baseline): CheXpert Plus 83.55 vs. 76.46 (+7.1%); MIMIC-CXR 71.92 vs. 64.74 (+7.2%); NIH CXR-14 66.42 vs. 60.79 (+5.6%); RSNA Pneumonia 86.26 vs. 84.63 (+1.6%); COVIDx CXR-4 76.37 vs. 66.90 (+9.5%).
  • In-domain adaptation (Table 3, CheXzero backbone, 100% data): CheXpert Plus (seen disease) 86.6 vs. 83.3 (LP); COVIDx CXR-4 (unseen disease) 78.1 vs. 73.4 (MMRL, +4.7%).
  • Zero-shot cross-dataset (Table 4, MIMIC→CheXpert Plus): MedBridge+MoE 79.94 AUC vs. MMRL 76.48 (+3.5%).
  • Retinal fundus generalization (ODIR-5K, Table 7): 72.58 AUC vs. 69.52 (MMRL).
  • MedGemma backbone (in-domain, Table 5): 85.83/76.88/89.68 AUC on CheXpert/MIMIC/RSNA.
  • Efficiency vs. LoRA: MQEncoder peak memory 3.9 GB vs. 10.3 GB; training 1219 vs. 1989 ms/step.
  • Ablation (Table 8): removing learnable queries causes the largest drop (−8.09% AUC CheXpert, −7.45% RSNA); removing MoE −5%; removing multi-view sampling −3.5%.

Limitations

  • Evaluated exclusively on chest radiography (five datasets) and one retinal fundus dataset; generalization to CT, MRI, pathology slides remains undemonstrated with quantitative results.
  • Multi-view sampling uses fixed sliding-window hyperparameters (N=11, 512×512 crops) without adaptation to modality-specific spatial scale, which may be suboptimal for non-CXR domains.
  • MoE performance scales with number and diversity of expert VLMs; in low-resource or proprietary settings, access to multiple large VLMs (e.g., LLaVA-7B, MedGemma-4B) simultaneously may be a barrier.
  • The query-fusion weight α is a fixed scalar hyperparameter rather than learned per sample or per disease class.
  • Code not yet publicly released (pending acceptance).

Relevance to Foundation Models in Medicine

MedBridge directly addresses a central challenge in medical foundation model research: how to efficiently bridge general-purpose VLMs to clinical tasks without the enormous cost of domain-specific pretraining. By keeping backbones frozen and adding only ~4% learnable parameters, it offers a practical alternative to full re-pretraining pipelines like CheXzero or MedCLIP, while remaining compatible with the latest general VLMs (Qwen3-VL, LLaVA, MedGemma). The query-driven MoE mechanism is particularly relevant as it enables plug-and-play integration of emerging medical VLMs without architectural constraints, making MedBridge a useful abstraction layer as the foundation model ecosystem continues to grow. For researchers tracking adaptation strategies in medical AI, the systematic comparison across three adaptation regimes and varying label budgets provides actionable benchmarks for both data-rich and data-scarce clinical settings.