Skip to content

Free Lunch in Medical Image Foundation Model Pre-training via Randomized Synthesis and Disentanglement

🕒 Published (v1): 2026-02-12 18:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RaSD is a synthetic-data-only pre-training framework for medical image foundation models (MIFMs) that generates image–label pairs on-the-fly from randomized Gaussian distributions, entirely eliminating real data dependency. By forcing models to disentangle anatomical structure from appearance via prototype-based learning, RaSD achieves competitive or superior performance to real-data-pretrained models across 56 downstream tasks and 6 imaging modalities.

Problem

MIFM pre-training is bottlenecked by the scarcity, heterogeneity, privacy constraints, annotation cost, and storage demands of large-scale real medical datasets. Existing synthetic approaches either rely on generative models that themselves require real data, or (like Anatomix) enforce appearance consistency in ways that suppress discriminative features and hurt modality-specific performance.

Method

RaSD has two coupled components:

Randomized synthetic data engine: Generates paired image–mask data entirely from Gaussian distributions in a streaming (online) fashion. The mask synthesis engine samples Gaussian heatmaps, applies deformation fields and space augmentation to produce anatomically plausible masks. The image synthesis engine samples region- and background-specific Gaussian noise to produce corresponding intensity images with varied appearance. No pre-storage is required; data is generated and consumed during training.

Prototype disentangling learning: Given a synthetic image and its pixel-level mask, a backbone (2D UNet CNN for RaSD-2D; Swin Transformer base in Swin UNETR for RaSD-3D) extracts dense features. Prototype vectors per semantic region are computed via mask pooling. A contrastive objective applies prototype separation (push inter-class prototypes apart) and prototype clustering (pull intra-class features toward their prototype), training the model to disentangle structural semantics from appearance, yielding transferable representations.

Scale: 1.2M synthetic 3D volumes (RaSD-3D, up to 653M parameters) and 9.6M synthetic 2D images (RaSD-2D).

Key Contributions

  • First demonstration that MIFMs pre-trained exclusively on rule-based synthetic data (zero real images) achieve broad competitive parity with real-data-pretrained models across 48 datasets and 56 tasks spanning CT, MRI, X-ray, ultrasound, fundus, and pathology.
  • Randomized Gaussian synthesis engine producing infinite, storage-free, auto-labeled training data for both 2D and 3D settings.
  • Prototype disentangling learning that decouples structural semantics from appearance, overcoming the appearance-suppression flaw of prior synthesis-based methods (Anatomix).
  • Online synthesis paradigm: eliminates pre-storage, reduces total training time by ~13% at scale, and reduces storage to 0 GB versus ~10 GB offline at 1000-sample scale.
  • Modality-agnostic pre-training that transfers across CT, MRI, X-ray, ultrasound, fundus, and pathology without any modality-specific real data.

Results

3D CT segmentation (vs. SuPreM, VoCo, Swin UNETR, Anatomix): - Best or tied-best on IRCADb, Covid-19-20, WHS-CT - >6% Dice improvement over scratch on TotalSegmentator, IRCADb, VerSe20 - Outperforms Anatomix on all 12 CT segmentation tasks

3D MRI segmentation: - Best performance on 3/5 tasks; consistently outperforms Anatomix, including cases where Anatomix falls below scratch on AMOS-MR

3D classification: - CC-CCII: 98.8% AUC (matches VoCo, beats SuPreM/Swin UNETR by up to 0.5%) - LUNA16: 98.1% AUC (within 0.6% of best) - MRNet: 80.9% AUC (beats scratch by 1.2%, within 2% of top)

2D X-ray (chest, skeletal, mammography): - MC set chest classification: +11.8% over scratch, surpasses LVM-Med (R50) and MedSAM - INbreast detection: outperforms Mammo-CLIP and MAMA by 4.1% - VinDr-Mammo detection: +1.4% over modality-specific FMs

Pathology segmentation (vs. PathSegmentor): - CoCaHis: +5.3% Dice over PathSegmentor - Janowczyk: +8.1% Dice over PathSegmentor - Outperforms all general-purpose FMs on 4/9 pathology datasets

Efficiency: - Online synthesis is 13% faster end-to-end than offline at 1000-image scale and requires 0 GB storage versus ~10 GB offline

Limitations

  • Rule-based Gaussian synthesis cannot capture rare disease manifestations or complex pathological/anatomical anomalies outside predefined rules.
  • Performance gap remains versus highly specialized modality-specific FMs (e.g., Mammo-CLIP, MAMA) on narrow domains, attributed to distribution mismatch.
  • Evaluation is primarily on public benchmark datasets; full validation on real-world clinical deployment data is not yet demonstrated.
  • The synthetic distribution is constrained to structures well-modeled by Gaussian mixtures; highly complex or irregular anatomy may be under-represented.

Relevance to Foundation Models in Medicine

RaSD directly addresses the central data bottleneck in MIFM development by establishing that high-quality, transferable representations can be learned without any real patient data, offering a privacy-preserving and infinitely scalable pre-training pathway. For researchers tracking foundation models in medicine, this is a significant existence proof: the prior assumption that large-scale real datasets are a prerequisite for competitive MIFMs is challenged across 6 modalities and 56 tasks. It also repositions synthetic data generation—previously relegated to data augmentation or domain adaptation—as a primary pre-training paradigm, opening a line of work exploring hybrid rule-based and generative synthesis. The modality-agnostic transferability result (e.g., synthetic pre-training improving MRI performance better than CT-specific real-data models) has direct implications for generalizable clinical AI development.