Skip to content

MEDFORM: A Foundation Model for Contrastive Learning of CT Imaging and Clinical Numeric Data in Multi-Cancer Analysis

🕒 Published (v1): 2025-01-22 23:56 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MEDFORM is a dual-stage multimodal pre-training framework that aligns volumetric CT representations with clinical numeric tabular data via cross-modal contrastive learning. It processes CT volumes through MIL-based slice aggregation and uses SimCLR-pretrained slice encoders before jointly learning CT–clinical embeddings. The approach outperforms unimodal and feature-concatenation baselines on cancer classification across three cancer types.

Problem

Annotated large-scale multimodal CT + clinical datasets are expensive and scarce. Existing SSL approaches for CT imaging operate unimodally and do not exploit the complementary signal available in routinely collected structured clinical data (demographics, disease history). Cross-modal contrastive learning between CT volumes and clinical numeric features has been underexplored compared to image–text pairing.

Method

MEDFORM applies a two-stage pre-training pipeline: 1. CT-SimCLR: A ResNet-50 is pre-trained from scratch using SimCLR on ~160K axial CT slices (NSCLC, breast, colorectal) to obtain slice-level embeddings. CT volumes are preprocessed with HU windowing, 1.0×0.8×0.8 mm resampling, and circular masking. 2. Multimodal alignment: Slice embeddings are aggregated into a patient-level volumetric representation via Attention-Based MIL (ABMIL). A 3-layer MLP encodes clinical numeric features. Both encoders are then aligned in a shared latent space using a symmetric cross-modal contrastive loss (SymCL, analogous to CLIP). The framework adapts TANGLE (originally for transcriptomics–pathology alignment) to the CT–tabular setting.

Downstream evaluation uses linear probing; few-shot evaluation varies k ∈ {1, 5, 10}.

Key Contributions

  • Dual pre-training strategy: SimCLR-based CT slice self-supervised pre-training followed by cross-modal contrastive alignment with clinical numeric data.
  • Application of ABMIL to aggregate 2D axial slices into patient-level CT representations without slice-level annotation.
  • Demonstration of cross-modal contrastive learning between CT and structured tabular clinical data across three distinct cancer types (NSCLC, breast, colorectal).
  • Few-shot learning evaluation showing robust performance with minimal labeled data.

Results

  • Breast cancer T-stage (ACRIN): Contrastive model AUROC 0.704 vs. unimodal 0.639 and feature-concatenation 0.630.
  • Colorectal bilobar disease (TCIA): Contrastive AUROC 0.534 vs. unimodal 0.483 and concatenation 0.515; ACC 0.658 vs. 0.508/0.508.
  • NSCLC histologic stage (TCIA): Contrastive AUROC 0.666 vs. unimodal 0.600 and concatenation 0.609.
  • Few-shot (TANGLE baseline, breast cancer): k=5 → AUROC 0.659, ACC 0.628; k=10 → AUROC 0.687, ACC 0.642.
  • Few-shot (NSCLC): k=10 → AUROC 0.582, ACC 0.630.
  • Contrastive learning consistently beats feature concatenation across all tasks.

Limitations

  • Absolute performance remains modest (AUROC 0.53–0.70), reflecting small and heterogeneous test cohorts with high variance (standard deviations often ≥0.10).
  • Few-shot evaluation is reported only against the TANGLE baseline, not against unimodal or concatenation few-shot variants.
  • Limited to three cancer types from TCIA; generalizability to other institutions or imaging protocols is untested.
  • MIL with binary classification is the authors' stated limitation; extension to multi-class staging is noted as challenging.
  • No ablation isolating the contribution of ABMIL pooling vs. the contrastive objective.
  • Clinical tabular features used are not described in detail, making reproducibility difficult.

Relevance to Foundation Models in Medicine

MEDFORM demonstrates a pathway for building CT foundation models without text supervision, instead leveraging routinely available structured clinical data as the paired modality—relevant as the field seeks annotation-efficient pre-training for 3D medical imaging. The TANGLE adaptation to CT + tabular data extends the vision–omics contrastive paradigm beyond pathology, broadening the template for non-language multimodal medical foundation models. The MIL-based volumetric encoding strategy addresses the practical constraint of 3D CT slice volume without requiring volumetric SSL architectures, an important efficiency consideration for scaling. This work positions clinical numeric data as a viable supervisory signal alongside radiology reports and genomics for CT representation learning.