Skip to content

Vision Foundation Models for Computed Tomography

🕒 Published (v1): 2025-01-15 18:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CT-FM is a 3D vision-only foundation model pre-trained on 148,000 CT scans from the NCI Imaging Data Commons using intra-sample contrastive learning (a modified SimCLR). It outperforms supervised and self-supervised baselines across whole-body segmentation, tumor segmentation, head CT triage, and content-based image retrieval, while demonstrating interpretable anatomically-structured embeddings.

Problem

Radiological AI is fragmented across task-specific models, and existing multimodal (vision-language) foundation models entangle visual representations with text, degrading fine-grained visual feature quality. 3D medical SSL pre-training is underexplored—prior work defaults to 2D slice-wise or video-style pre-training that lacks 3D spatial semantics—leaving a gap for a dedicated, native-3D, vision-centric CT encoder.

Method

CT-FM uses intra-sample contrastive learning built on a modified SimCLR framework. The key modification: instead of treating other batch elements as negatives, patches sampled from within the same scan serve as negative pairs. This forces the model to discriminate anatomical sub-volumes at a fixed concept scale (24×128×128 voxels, ≈12.8×12.8×6.4 cm, roughly the size of a human heart). Two augmented views per patch (random crop/resize, histogram shifting, affine, intensity variation) are passed through a SegResNet encoder (512-d latent, then projection head), optimized with normalized temperature-scaled cross-entropy (τ=0.1). Pre-training ran for 500 epochs on 148,000 CT scans (resampled to 3 mm slice thickness, 1 mm in-plane) across 4× NVIDIA Quadro RTX 8000 GPUs for 24 days. Optimal checkpoint was selected at epoch 449 via few-shot segmentation on TotalSegmentator validation.

Key Contributions

  • Intra-sample contrastive objective: patch-level negatives drawn from within the same scan rather than across the batch, promoting fine-grained 3D anatomical discrimination.
  • Large-scale 3D vision-only pre-training: 148,000 CT scans (69 cohorts, cancer-enriched), entirely label-agnostic.
  • Broad task evaluation: whole-body segmentation (117 labels), multi-site tumor segmentation, head CT triage (in- and out-of-distribution), content-based image retrieval, and zero-shot semantic/embedding analysis.
  • Interpretability analyses: anatomical clustering (tSNE), semantic concept search with organ centroid distance (OCD) metric, PCA color-space visualization, occlusion-based saliency mapping.
  • Test-retest robustness validated on the RIDER dataset.
  • Full open-source release of model weights, code, and pre-training dataset.

Results

Whole-body segmentation (TotalSegmentator, 117 labels, Dice): - CT-FM: 0.8981 vs. SuPREM: 0.8695, Auto3DSeg: 0.882, VISTA3D: 0.893, architectural baseline: 0.8959 - CT-FM wins on 73.5% of individual labels vs. baseline; 91.5% vs. SuPREM - CT-FM (Merlin split): 0.9058 vs. Merlin: 0.862

Tumor segmentation (MSD, Auto3DSeg backbone): - Hepatic: Dice 0.696 (CT-FM) vs. 0.681 (baseline); ASD 2.8 vs. 3.6 mm - Lung: Dice 0.609 vs. 0.532; ASD 48.3 vs. 62.1 mm - Pancreatic: Dice comparable (0.475 vs. 0.482), ASD 7.8 vs. 13.8 mm

Head CT triage (SinoCT): - CT-FM F1: 0.776, AUC: 0.836 vs. baseline F1: 0.754, AUC: 0.802; SuPREM best at F1: 0.798, AUC: 0.868 - Zero-shot CQ500: CT-FM F1: 0.754, AUC: 0.794; SuPREM F1: 0.745, AUC: 0.793 (CT-FM exceeds SuPREM OOD)

Image retrieval (OrganMNIST3D, k=3): - CT-FM AP: 0.932, HR: 0.968, F1: 0.944 vs. SuPREM AP: 0.923, HR: 0.959, F1: 0.935

Semantic concept search (heart OCD): - CT-FM: 5.61 ± 5.44 cm; SuPREM: 23.44 ± 7.06 cm; 97.7% vs. 81.9% patch-structure co-location

Limitations

  • No cross-validation or ensembling due to compute constraints; single-run results throughout.
  • Pre-training design space explored only via intuition-guided ablations; broader hyperparameter search was infeasible.
  • Diagnostic and prognostic quantification tasks (e.g., staging, survival prediction) not evaluated.
  • Head CT anatomy was rare in the pre-training corpus, yet CT-FM was applied to head triage; domain gap partially limits performance.
  • Qualitative analyses on a random scan subset only; not exhaustive.
  • Pre-training dataset is cancer-cohort-heavy (NLST-dominant), which may bias representations.

Relevance to Foundation Models in Medicine

CT-FM exemplifies the emerging design principle that modality-specific, vision-only pre-training on large unlabeled 3D corpora can match or exceed multimodal (vision-language) and supervised foundation models on dense visual tasks, directly challenging the assumption that text supervision is necessary for generalization. The intra-sample contrastive strategy addresses a fundamental mismatch between natural-image SSL assumptions and the high inter-scan homogeneity of medical volumes—a transferable insight for MRI, PET, and ultrasound pre-training. The open-sourced model (weights + 148k-scan dataset) positions CT-FM as a practical backbone for the broader GMAI ecosystem, particularly for retrieval-augmented diagnosis and low-annotation clinical deployment.