Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development¶

🕒 Published (v1): 2026-03-29 00:46 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents the largest survey to date of medical imaging datasets, cataloging 1,000+ open-access collections across modalities, tasks, and anatomies from 2000–2025. It diagnoses the landscape as fragmented, modest in scale, and unevenly distributed, then proposes a Metadata-Driven Fusion Paradigm (MDFP) and an interactive discovery portal to systematically consolidate disparate datasets into larger training corpora for medical foundation models.

Problem¶

Medical imaging foundation models are severely data-constrained: most public datasets contain only thousands of images versus the billions used to train general-domain models (e.g., SA1B, LAION-5B). Data is scattered across narrowly scoped, isolated collections skewed toward a few modalities (CT, MRI) and organs (brain, lung), causing modality-, task-, and anatomy-specific biases. Prior surveys lacked subject/image-level statistics, omitted recent large-scale datasets, and provided no systematic framework linking dataset characteristics to foundation model development requirements.

Method¶

The authors compile 1,000+ open-access datasets from major repositories (TCIA, Grand Challenge, etc.) via deduplication, manual landing-page verification, and metadata normalization. They organize datasets via a four-level taxonomy: imaging dimensionality (2D/3D/video) → modality → task → anatomical region. Gap analysis identifies underrepresented axes. The Metadata-Driven Fusion Paradigm (MDFP) then operates in four phases: 1. Metadata Harmonization — normalize heterogeneous metadata fields across datasets. 2. Semantic Alignment — reconcile label spaces across datasets sharing modality or task. 3. Fusion Blueprints — generate integration recipes (goal-conditioned or modality/task-driven) to merge data silos into coherent corpora. 4. Dataset Indexing & Community Sharing — release unified structured table, Python toolkit, and an interactive browser portal enabling fine-grained search, statistical analysis, and automated integration.

Key Contributions¶

Comprehensive catalog of 1,000+ open-access medical imaging datasets (2000–2025) with standardized metadata covering modality, task, anatomy, annotations, and licenses.
Novel four-level dataset taxonomy (dimensionality → modality → task → anatomy) aligned with foundation model architectural and training needs.
MDFP: a systematic four-phase workflow for merging fragmented datasets into larger, more coherent training resources.
Quantitative gap analysis exposing underrepresented modalities (PET, mammography, endoscopy), anatomical regions (foot, shoulder, blood, bowel), and tasks (registration, tracking, localization, VQA).
Interactive discovery portal enabling end-to-end automated dataset search and integration, plus a released merged large-scale dataset and Python toolkit.

Results¶

Scale: 502 labeled 2D datasets (≈1.4M 2D CT slices alone from 39 sub-datasets ranging from 2 to 874,000 images); 3D and video datasets significantly smaller in absolute image count.
Modality distribution: Pathology dominates by image count (WSI patch multiplication); X-ray, CT, and MRI follow; MRI accounts for ~10.4% of total images.
Task distribution: Classification and segmentation dominate labeled data; generation surged markedly post-2023; registration, tracking, and VQA remain severely underrepresented.
Anatomy distribution: Brain, lung, liver, breast, retina, and full-body datasets substantially outnumber all other regions; foot, blood, heart, shoulder, and forearm are sparsely covered.
Temporal trends: Clear inflection in dataset release after 2012 (deep learning era) and a second surge after 2023 (foundation model era); largest public datasets include AbdomenAtlas (1.5M 2D CT images, 5,195 3D volumes) and CT-RATE (25,692 3D chest CT scans from 21,304 patients).
No model training benchmarks or accuracy numbers are reported; this is a survey and infrastructure paper.

Limitations¶

Medical imaging datasets remain orders of magnitude smaller than natural-image or language corpora (trillions of tokens vs. thousands–millions of medical images), and this gap is unlikely to close via curation alone.
3D volumetric and video datasets are comparatively scarce and grow slowly due to acquisition cost, storage requirements, and annotation complexity.
Multimodal medical datasets (cross-modality paired data) are identified as critically scarce, constraining multi-modal foundation model development.
Gap analysis operates at a coarse anatomical granularity; finer-grained anatomical systems are unevenly annotated across public sources, limiting deeper analysis.
MDFP is presented as a paradigm with a case study but without large-scale empirical validation demonstrating downstream foundation model performance gains from fused corpora.
Existing data skew (long-tail distribution) risks being amplified rather than corrected if fusion is applied naively without balanced sampling.

Relevance to Foundation Models in Medicine¶

This survey directly addresses the primary bottleneck for scaling medical foundation models: the absence of a systematic view of what training data exists, where gaps lie, and how to consolidate fragmented public datasets. The MDFP provides a principled, reproducible method for assembling heterogeneous corpora that parallels data-curation practices behind general-domain foundation models. For researchers tracking this topic, the gap analysis and taxonomy serve as a planning instrument for future dataset creation priorities, while the portal offers practical tooling for rapid data discovery and integration. The work also quantifies the structural mismatch between medical and general-domain data scales, framing dataset consolidation—not just model architecture—as a first-order research problem.