Skip to content

M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model

🕒 Published (v1): 2026-04-10 04:06 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

M-IDoL is a self-supervised medical foundation model that addresses "information ambiguity" in unified multimodal pre-training by decomposing the contrastive learning objective into two entropy-based terms: one that pushes modality representations apart (inter-modality specificity) and one that pulls same-modality views together within dedicated subspaces (intra-modality diversity). Pre-trained on 1.15M unlabeled images across five imaging modalities, it outperforms 20 existing foundation and modality-specific models on 21 downstream clinical tasks.

Problem

Existing medical foundation models (MFMs) apply a uniform contrastive learning (CL) objective across heterogeneous imaging modalities (X-ray, fundus, OCT, dermoscopy, pathology). This forces a single embedding space to accommodate all modalities, causing the MI-maximization objective to reduce both augmentation noise and cross-modality semantic variance — uniformly maximizing inter-modality redundancy. The result is "information ambiguity": representations lose modality-specific discriminative detail (degraded inter-modality specificity) and fine-grained intra-modality structure (degraded intra-modality diversity), hurting downstream clinical task performance.

Method

M-IDoL uses a DINO-style siamese encoder (Swin-B student/teacher) and reformulates the pre-training objective via information decomposition. The core insight is that directly subtracting the trivariate mutual information term I(X;Y;Z) (where Z = other-modality representations) from the standard bivariate CL objective is intractable. Instead, the authors decompose this into an entropy-based form:

max H(X|Z) − H(X|Y,Z)

This yields two concrete objectives implemented via a Mixture-of-Experts (MoE) projector with N experts (one per modality), operating without explicit modality labels:

  1. Inter-modality entropy maximization (H(X|Z)) — a routing-consistency loss (L_route) using cross-entropy between student/teacher routing probabilities over M augmented views, with Sinkhorn-Knopp normalization to enforce balanced expert assignment. Same-modality images route to the same expert; different modalities route to different experts, pushing their representations into separated subspaces.

  2. Intra-modality uncertainty minimization (H(X|Y,Z)) — an InfoNCE-based intra-modality contrastive loss (L_cst) that pulls augmented views of the same image together within each expert's subspace, reducing augmentation noise without conflating cross-modality variance.

The final loss is L_route + L_cst. At inference, only the teacher encoder is retained; MoE projectors are discarded.

Key Contributions

  • Formal identification of "information ambiguity" in unified multimodal CL pre-training as uniform maximization of inter-modality redundancy.
  • Information-theoretic decomposition of the trivariate MI objective into tractable entropy terms (H(X|Z) − H(X|Y,Z)).
  • MoE projector design that discovers latent modality subspaces without modality labels, implementing both specificity and diversity objectives.
  • A 1.15M-image pre-training corpus across five modalities with a strict no-overlap policy to downstream evaluation sets.
  • State-of-the-art results on 21 downstream datasets against 20 baselines (modality-specific and unified MFMs).

Results

  • vs. Unified MFMs (LVM-Med, Unimiss+, UniMed, MedCoss, CoSMIC): M-IDoL achieves >6% average improvement over Unimiss+, LVM-Med, and UniMed; >4% improvement over MedCoss and CoSMIC in OCT and pathology tasks; statistically significant (p < 0.05) on most of 21 datasets.
  • vs. Modality-specific models: Significantly surpasses RETFound (pretrained on 1.6M fundus images) on fundus tasks; surpasses MIRAGE (260K OCT images) on OCT tasks; competitive with Panderm (2.1M dermoscopy images) on dermoscopy.
  • Selected headline numbers (AUC%, from Table 3):
  • APTOS (fundus): 95.19 vs. 94.36 (RETFound, second best)
  • OCTID (OCT): 99.34 vs. 99.24 (MIRAGE)
  • Mitosis (pathology): 95.62 vs. 93.34 (UNI)
  • HAM10000 (dermoscopy): 97.96 vs. 97.13 (Panderm)
  • RSNA (X-ray): 93.23 vs. 91.68 (AFiRe)
  • Ablation (Table 2): MoE alone provides no gain; adding L_route yields substantial improvement; adding L_cst further improves across all five sampled datasets.
  • t-SNE visualization: M-IDoL achieves sharper inter-modality cluster separation and finer intra-modality sub-clustering than joint pre-training, MedCoss, and CoSMIC.

Limitations

  • Pre-training is limited to five imaging modalities; generalization to other medical modalities (CT, MRI, ultrasound) is not demonstrated.
  • Swin-B backbone and 100-epoch pre-training on 2× A6000 GPUs: the computational regime is modest compared to large-scale VLMs or billion-parameter MFMs, and scalability to larger models/datasets is untested.
  • The number of MoE experts must match the number of modalities; adding new modalities likely requires re-pre-training from scratch.
  • Downstream evaluation uses linear probing for pathology but full fine-tuning for other modalities — this asymmetry complicates direct cross-modality comparison.
  • Statistically non-significant improvements on NIH (p=0.099) and ISIC2016 (p=0.328) vs. CoSMIC, and on MHIST and SIIM-ACR (n.s.), suggesting limits in some domains.
  • No evaluation of few-shot or zero-shot transfer, which is increasingly the standard test for foundation model generalization.

Relevance to Foundation Models in Medicine

M-IDoL directly addresses a fundamental design flaw in unified medical foundation models — that standard contrastive pre-training inadvertently collapses inter-modality distinctions — and provides an information-theoretic remedy. For researchers tracking MFMs in medicine, this paper offers a principled framework (information decomposition) for reasoning about representation quality in multimodal settings, extending beyond prior work that relied on continual learning (MedCoss, CoSMIC) or modality-specific pre-training to achieve specificity. The MoE-projector mechanism — modality-agnostic, discarded at inference — is a lightweight and practical contribution that could be incorporated into future large-scale MFM pipelines. The breadth of evaluation (21 datasets, 5 modalities, 20 baselines) sets a new benchmark for comprehensive MFM comparison in the vision-only medical pre-training paradigm.