Skip to content

Data Scaling Laws for Radiology Foundation Models

🕒 Published (v1): 2025-09-16 08:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically measures data scaling laws for chest X-ray foundation models by continually pretraining MedImageInsight (MI2/CLIP-style) and RAD-DINO (DINOv2-style) on up to 3.5M institution-specific CXRs under controlled compute. CLIP-style MI2 scales ~3× faster than RAD-DINO for findings classification, while RAD-DINO dominates segmentation of elongated structures (lines & tubes). Even 30k in-domain samples can surpass open-weight baselines on findings classification, motivating center-specific continual pretraining.

Problem

Medical imaging foundation models are trained on datasets orders of magnitude smaller than web-scale counterparts, yet no controlled study has characterized how pretraining data volume drives performance across diverse CXR tasks. Prior comparisons of CLIP vs. DINOv2 paradigms in medicine are confounded by heterogeneous training data and compute budgets, and evaluations focus narrowly on radiological findings while neglecting clinically important structures like lines and tubes (l&t).

Method

Starting from open-weight MI2 (CLIP/UniCL, DAViT backbone) and RAD-DINO (DINOv2, ViT-B) checkpoints, the authors perform continual pretraining on five strictly nested subsets of INST-CXR-BENCH (30k / 50k / 100k / 1M / 3.5M image-report pairs) using fixed wall-clock time on 4×8 H100 nodes. Three MI2 variants are compared: CLIP (image-report only), UniCL (image-report + GPT-extracted tube-presence labels). Downstream evaluation uses frozen encoders with 4-layer feature concatenation + attention pooling + linear classifier/segmentation head, avoiding CLS-token-only evaluation. Tasks span: (1) findings classification (19 classes, INST-CXR-BENCH-FIND-CLASS, 2M frontal CXRs; holdout VinDR); (2) l&t presence classification (11 tube types, INST-CXR-BENCH-TUBE-CLASS, 1M CXRs); (3) l&t segmentation (RANZCR-CLiP, Hausdorff distance); (4) radiology report generation via MAIRA-2 (Vicuna-13B LLaVA). Power-law fits f(x) = αx^k are applied to classification scaling curves. The internal test set contains 400k samples to capture long-tail findings.

Key Contributions

  • First controlled, compute-matched scaling law study comparing CLIP vs. DINOv2 paradigms for CXR vision encoders up to 3.5M in-domain samples.
  • Demonstrates MI2 scales ~3× faster than RAD-DINO on findings classification (slopes k=0.012 vs k=0.004); RAD-DINO dominates l&t segmentation and maintains advantage at full scale.
  • Shows UniCL with GPT-extracted tube labels significantly improves MI2 on l&t tasks (closes gap to RAD-DINO on segmentation), yielding an unexpected synergistic benefit beyond report-only supervision.
  • Establishes that 30k–100k in-domain samples suffice to significantly surpass open-weight foundation model baselines for findings classification.
  • Introduces "curve-continuity features" (CCF) as a task category and Incorrect Placement F1 as a novel clinical efficacy metric for misplaced l&t detection in report generation.
  • Introduces INST-CXR-BENCH: 4M de-identified CXR-report pairs from Mayo Clinic (3.1M studies, 2013–2023), with GPT-4o-structured labels and 400k-sample test set.
  • First systematic layer-wise analysis of CXR vision encoders, showing layer choice substantially affects downstream performance.

Results

  • Findings classification (INST-CXR-BENCH-FIND-CLASS): MI2 (3.5M, reports) avg AUPRC 52.8 vs MI2 OWM 50.3 vs RAD-DINO OWM 49.6; MI2 outperforms RAD-DINO significantly at ≥100k. Rib fracture: MI2 3.5M shows +6% AUPRC over OWM.
  • Findings classification (VinDR holdout): MI2 reports+tubes achieves avg AUPRC 60.7 vs RAD-DINO OWM 60.2; RAD-DINO shows reversed scaling (catastrophic forgetting at scale).
  • L&t presence classification (INST-CXR-BENCH-TUBE-CLASS): MI2 reports+tubes avg AUPRC 92.4 vs RAD-DINO 92.1 vs MI2 OWM 90.5 at 3.5M; RAD-DINO consistently matches or leads report-only MI2.
  • L&t segmentation (RANZCR-CLiP, Hausdorff distance, lower=better): RAD-DINO 3.5M avg HD 13.1 vs RAD-DINO OWM 21.4 vs MI2 reports+tubes 15.6 vs MI2 OWM 23.1; RAD-DINO 3.5M significantly outperforms all others.
  • Report generation (MAIRA-2): MI2 significantly outperforms RAD-DINO on CheXbert Macro F1-14; RAD-DINO significantly outperforms report-only MI2 on Incorrect Placement F1; adding tube labels to MI2 narrows this gap. Both models require ≥1M samples to surpass OWM baselines on report metrics.
  • Data efficiency: 30k in-domain samples sufficient to significantly surpass MI2 OWM on findings classification; 100k sufficient for RAD-DINO.

Limitations

  • Single-institution dataset (Mayo Clinic); demographic skew (87% White) limits generalizability claims.
  • Compute and model size are held fixed (ViT-L/ViT-B scale); cannot disentangle data vs. model size scaling interactions.
  • GPT-4o-extracted labels introduce noise from inter-reader variation, inaccurate source reports, and extraction errors, affecting label quality for less prevalent findings.
  • VinDR holdout is too small (9k images) to yield reliable cross-domain scaling trends; RANZCR-CLiP segmentation benchmark (17k) similarly limits trend estimation below 100k.
  • Report generation experiments train only one MAIRA-2 model per configuration (cost-constrained), relying on bootstrap CIs rather than true replication.
  • Power-law behavior is observed in a limited data range; saturation and emergent/inverse scaling effects (noted in literature) may appear beyond studied scales.
  • Modality scope is chest X-ray only; conclusions may not transfer to CT, MRI, or other imaging domains.

Relevance to Foundation Models in Medicine

This work provides the first rigorous, controlled empirical characterization of how pretraining data volume translates to downstream performance for two dominant CXR foundation model paradigms, directly informing how institutions should invest in domain-specific continual pretraining. The finding that CLIP and DINOv2 paradigms have complementary strengths—CLIP for semantic classification, DINOv2 for spatially precise segmentation of elongated structures—has immediate architectural implications for multi-task CXR foundation model design. The result that UniCL with structured supervision bridging both paradigms' weaknesses supports the trend toward hybrid pretraining objectives in medical foundation models. The demonstration that center-specific adaptation with as few as 30k samples can beat general open-weight models is a practical prescription for deployment of medical foundation models in resource-constrained institutional settings.