Skip to content

Benchmarking CXR Foundation Models With Publicly Available MIMIC-CXR and NIH-CXR14 Datasets

🕒 Published (v1): 2025-12-03 12:55 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks two proprietary CXR foundation model encoders — Google's CXR-Foundation (ELIXR v2.0) and Microsoft's MedImageInsight — on MIMIC-CXR and NIH ChestX-ray14 using frozen embeddings with lightweight LightGBM classifiers. MedImageInsight marginally outperforms CXR-Foundation on most tasks while CXR-Foundation shows stronger cross-dataset consistency. The study establishes reproducible public baselines for comparing CXR embedding models.

Problem

Systematic, reproducible comparisons between recent CXR foundation model encoders are lacking. Most prior work benchmarks individual models in isolation; the relative performance and cross-dataset generalization of CXR-Foundation vs. MedImageInsight on standard public datasets (MIMIC-CXR, NIH ChestX-ray14) is unknown.

Method

Frozen embeddings are extracted from both pretrained encoders without any fine-tuning. CXR-Foundation produces 32×768 token features that are mean-pooled; MedImageInsight produces 1024-dimensional vectors. Images are standardized (1024×1024, z-score normalized, MONOCHROME2) with patient-wise 80/20 splits and five augmented views per training image. LightGBM classifiers are trained on balanced 2,000-image subsets (1,000 positive/negative per label) for four pathologies: Atelectasis, Edema, Effusion, Opacity. Evaluation uses 5-fold patient-wise cross-validation with mean AUROC and F1 (95% CI). UMAP + k-means clustering (cosine distance, Silhouette-optimized k) is applied to MedImageInsight embeddings for qualitative structure analysis.

Key Contributions

  • First head-to-head benchmark of CXR-Foundation and MedImageInsight under identical preprocessing and classifiers on two major public CXR datasets.
  • Reproducible pipeline (public datasets, open-source tools, single A100 GPU) establishing reference AUROC/F1 baselines for four thoracic disease labels.
  • Unsupervised embedding structure analysis via UMAP/k-means showing MedImageInsight embeddings encode disease-specific geometry coherent with downstream task performance.

Results

  • MedImageInsight generally achieves higher AUROC and F1; both models exceed AUROC 0.90 on most tasks.
  • Effusion (MIMIC): MedImageInsight 0.958 vs. CXR-Foundation 0.941 AUROC; F1 0.906 vs. 0.877.
  • Effusion (NIH): Equal AUROC at 0.901; F1 0.828 vs. 0.826.
  • Edema (MIMIC): CXR-Foundation edges ahead at AUROC 0.924 vs. 0.918; F1 nearly equal (0.847 vs. 0.841).
  • Opacity (NIH): CXR-Foundation higher at AUROC 0.955 vs. 0.922; F1 0.889 vs. 0.851.
  • Atelectasis (NIH): MedImageInsight 0.863 vs. CXR-Foundation 0.822 AUROC.
  • CXR-Foundation shows narrower performance variance across MIMIC vs. NIH splits, indicating cross-dataset stability.
  • UMAP of MedImageInsight embeddings: Effusion yields distinct positive/negative cluster separation; Opacity shows overlap, consistent with quantitative performance.

Limitations

  • Only two foundation models evaluated; no fine-tuned or open-source models (e.g., CheXzero, BioViL) included for comparison.
  • Restricted to frontal (PA/AP) views; lateral projections excluded.
  • Balanced sampling (1,000 pos/neg per label) may not reflect real-world prevalence distributions.
  • LightGBM probing only; no assessment of fine-tuning or zero-shot transfer.
  • MIMIC-CXR and NIH ChestX-ray14 have limited demographic diversity, risking undetected bias.
  • Results may not generalize to other architectures, modalities, or clinical sites.

Relevance to Foundation Models in Medicine

This work directly addresses a practical gap in the medical foundation model literature: how to fairly compare large proprietary CXR encoders without fine-tuning access. The frozen-embedding + lightweight-classifier protocol offers a generalizable evaluation framework applicable to other imaging modalities and foundation models beyond CXR. For researchers tracking this topic, the finding that embedding dimensionality and alignment (MedImageInsight's 1024-d vs. CXR-Foundation's 32×768 mean-pooled) correlates with downstream task performance is an actionable architectural signal. The reproducible public baselines also provide a needed reference point as the field moves toward multimodal clinical integration.