Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM)¶

🕒 Published (v1): 2025-06-30 18:50 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MerMED-FM is a vision-only, self-supervised foundation model trained on 3.3 million images spanning seven modalities and ten-plus specialties, unified via a memory-augmented teacher-student framework. It achieves mean AUROC of 0.935 across all modalities, outperforming or matching both general-domain (DINO) and the leading multispecialty baseline (BiomedCLIP), while also competing with single-modality specialists on most tasks. The model demonstrates strong data efficiency, retaining near-peak performance with as little as 10–50% of fine-tuning data.

Problem¶

Existing medical imaging FMs are predominantly single-modality and single-specialty (e.g., RETFound for retinal OCT/CFP, Merlin for CT, UNI for pathology), requiring separate pipelines per modality and failing to reflect real clinical workflows where multi-modal integration is routine. Prior attempts at multimodal/multi-disease models have shown inconsistent accuracy, and most rely on large labeled datasets or image-text pair supervision.

Method¶

MerMED-FM uses a dual-network teacher-student architecture based on Vision Transformers (ViT-B), trained with self-supervised learning (SSL) on unlabeled images. The core innovation is a dynamic memory module (size K=65,536, FIFO update) that stores compact cross-modal representations from prior training iterations. During training, 12 augmented views per image (2 global at 224×224, 10 local at 96×96) are encoded by student and teacher encoders; representations are compared against memory-stored concepts via non-parametric similarity distributions, enforcing cross-view and cross-modal consistency and preventing catastrophic forgetting. Balanced modality-and-specialty sampling within each batch prevents domain dominance. Downstream adaptation fine-tunes only the student ViT-B encoder plus a task-specific MLP head. Pre-training used 8×H100 GPUs for ~3 days.

Key Contributions¶

Single unified vision-only FM covering CT, CXR, ultrasound, histopathology, CFP, OCT, and dermoscopy without requiring image-text pairs or language input
Memory-augmented SSL framework with FIFO dynamic memory to prevent catastrophic forgetting during heterogeneous multi-modal training
Balanced modality-specialty sampling to prevent data-imbalance skewing toward dominant domains
Demonstrated data efficiency: competitive performance with 10% of fine-tuning data across multiple modalities
Validation on both 25 public datasets and 9 real-world clinical datasets from a Singapore tertiary hospital cluster

Results¶

Overall mean AUROC 0.935 vs. BiomedCLIP 0.919, DINO 0.933
CT: mean AUROC 0.975; outperforms Merlin (specialist CT FM) by 3.61% on lung cancer (IQ-OTHNCCD, p<0.01, Cohen's d=3.51); COVID-19 CT AUROC 0.999 vs. Merlin 0.987
CXR: mean AUROC 0.908; outperforms RadDINO by 12.2% on pneumonia (p<0.01, Cohen's d=20.0) and 3.45% on pneumothorax
Ultrasound: mean AUROC 0.951; surpasses USFM (0.939) on BUSI dataset (AUROC 0.973 vs. 0.966, p=0.033)
Histopathology: mean AUROC 0.999; statistically equivalent to UNI (trained on 100M patches) despite far fewer images
OCT: mean AUROC 0.988; outperforms RETFound by 3.92% on OCTID (p<0.001, Cohen's d=10.6)
CFP: mean AUROC 0.894; outperforms RETFound by 2.14% on glaucoma (Glaucoma Fundus, Cohen's d=7.40)
Dermoscopy: mean AUROC 0.931; closely trails specialist PanDERM (0.81% behind on Dermnet)
Data efficiency: with 50% data, AUROC 0.926 vs. full-data 0.948; at 10% fine-tuning data, highest US F1 (0.708) vs. USFM (0.662)

Limitations¶

Does not outperform single-modality specialists in pathology (UNI) and dermatology (PanDERM)
Processes only 2D slices; no volumetric imaging support (CT/US volumes not exploited as 3D)
Has not been evaluated for synergistic within-patient multimodal reasoning (e.g., joint CT + histopathology diagnosis)
Clinical applicability tested at only one hospital cluster (Singapore); generalizability to other health systems unvalidated
Vision-only design precludes leveraging clinical text context that image-text models (BiomedCLIP, MedImageInsight) exploit
No segmentation or report-generation capability evaluated

Relevance to Foundation Models in Medicine¶

MerMED-FM directly advances the field's core aspiration of a single general-purpose medical vision encoder by demonstrating that SSL with memory-based regularization can overcome catastrophic forgetting across radically heterogeneous modalities—a problem that has previously forced the field toward specialist silos. Its vision-only design is a deliberate architectural choice that makes it more deployment-ready in real hospital triaging pipelines where text annotations are unavailable at inference time, contrasting with the dominant image-text contrastive paradigm (BiomedCLIP, BIOMEDICA). The data-efficiency findings are particularly relevant to low-resource clinical settings, suggesting that broad multimodal pre-training confers representation quality that transfers with minimal labeled fine-tuning data. For researchers tracking medical FMs, MerMED-FM sets a new precedent for how cross-specialty generalization can be achieved without sacrificing per-modality performance relative to specialist models.