Skip to content

Feature Quality and Adaptability of Medical Foundation Models: A Comparative Evaluation for Radiographic Classification and Segmentation

🕒 Published (v1): 2025-11-12 21:16 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks eight vision encoders (six medical-domain, two general-domain FMs) on chest X-ray classification and segmentation via linear probing and fine-tuning. Medical-domain pre-training consistently yields superior off-the-shelf feature quality, but pre-trained features are insufficient for localizing subtle pathologies like pneumothorax without significant fine-tuning. A supervised end-to-end baseline (SegFormer) remains competitive or best for segmentation, and text-image alignment is not necessary for strong downstream performance.

Problem

It is unclear how pre-training domain (medical vs. general), paradigm (text-guided contrastive vs. image-only self-supervised vs. supervised multi-source), and architecture (ViT vs. hierarchical Swin/DaViT) affect embedding quality and adaptability for specific radiology tasks. Current evaluations focus on classification and do not probe fine-grained spatial features needed for segmentation.

Method

Eight Transformer-based vision encoders—Ark+ (Swin-Large, supervised multi-source), BiomedCLIP (ViT-B, CLIP), CheXagent (SigLIP-Large), MedImageInsights (DaViT, UniCL), MedSigLIP (SigLIP2-So400m), RAD-DINO (ViT-B/14, image-only DINO), DINOv2 (ViT-B/14, general), SigLIP2 (general)—plus a SegFormer trained from scratch as baseline, are evaluated on two tasks: binary classification (pneumothorax AUROC, cardiomegaly AUROC) and segmentation (pneumothorax Dice, cardiac boundary Dice). Two evaluation stages: (1) linear probing — frozen encoder + single linear head trained on SIIM-ACR (n=10,675 CXR) and a private cardiomegaly cohort (n=5,000 CXR); (2) fine-tuning — full encoder + head with differential learning rates (1e-7 encoder, 1e-4 head). Hierarchical models evaluated at 16×16 vs. 32×32 effective patch sizes. Subgroup analysis stratifies small pneumothorax cases by chest-tube presence to detect shortcut learning. Statistical comparisons use Friedman + Nemenyi post-hoc tests; five-fold patient-level cross-validation (64-16-20 split).

Key Contributions

  • Systematic two-stage (linear probing + fine-tuning) evaluation isolating feature quality from adaptability across classification and segmentation.
  • Demonstrates domain gap quantitatively: medical FMs outperform general FMs by up to 0.10 AUROC in linear probing for classification.
  • Shows pre-trained features are insufficient for subtle pathology segmentation (pneumothorax linear probing: 0.219–0.433 Dice across all FMs) but strong for salient anatomy (cardiac linear probing: 0.890–0.942 Dice).
  • Subgroup analysis reveals FMs use chest-tube presence as a shortcut for pneumothorax classification, failing without tubes (near-random sensitivity) while segmentation is poor regardless.
  • Shows image-only (RAD-DINO) and label-supervised (Ark+) encoders match or exceed text-guided VLMs, challenging the necessity of image-text alignment.
  • Documents practical adoption barriers: poor preprocessing documentation, embedding extraction complexity, closed-source inaccessibility.

Results

Pneumothorax classification (AUROC, linear probing): - Medical FMs: 0.877 (BiomedCLIP) – 0.964 (Ark+) - General FMs: DINOv2 0.870, SigLIP2 0.871 - SegFormer baseline: 0.890; Ark+ significantly outperforms SegFormer after fine-tuning

Cardiomegaly classification (AUROC, linear probing): - Medical FMs: 0.892–0.945; General FMs: 0.842–0.866 - MedImageInsights fine-tuning decreased performance (0.945→0.937, p=0.025)

Pneumothorax segmentation (Dice, linear probing → fine-tuning): - All FMs poor at linear probing: 0.219–0.433 - Fine-tuning critical: SegFormer best at 0.478; MedImageInsights 0.476 after fine-tuning - Small pneumothorax subgroup: SegFormer 0.308 (no chest tube), 0.248 (chest tube present)

Cardiac segmentation (Dice, linear probing → fine-tuning): - Linear probing: 0.890–0.942; fine-tuning convergence: 0.908–0.961 - MedImageInsights highest at 0.961, matching SegFormer

Shortcut finding: On small pneumothorax with chest tube, MedImageInsights linear probing sensitivity 0.967; without chest tube, near-random across all models.

Limitations

  • Evaluation restricted to chest radiographs (two findings); generalizability to other modalities/pathologies untested.
  • CheXagent had prior exposure to the SIIM-ACR pneumothorax dataset during pre-training, confounding comparison.
  • Third-party implementation of MedImageInsights may not reflect intended performance.
  • Five-fold cross-validation may lack power to detect subtle inter-model differences.
  • Private cardiomegaly dataset limits reproducibility.
  • No evaluation of zero-shot or text-based inference; conclusions on text alignment apply only to adaptation settings.

Relevance to Foundation Models in Medicine

This study is directly relevant to the question of how to select and deploy medical FMs in practice: it quantifies the domain-pretraining advantage while exposing its limits for dense prediction tasks, a setting increasingly important for clinical decision support. The finding that hierarchical multi-scale architectures (Ark+, MedImageInsights) systematically outperform isotropic ViTs for segmentation has architectural design implications for next-generation medical FMs. The chest-tube shortcut result is a concrete example of the FM reliability problem—models that appear strong by aggregate metrics exploit spurious correlates rather than pathology-relevant features—directly informing evaluation methodology for the field. The result that image-only self-supervised encoders (RAD-DINO) and supervised multi-source models (Ark+) rival text-aligned VLMs challenges the prevailing design assumption that vision-language alignment is necessary for high-quality medical vision representations.