Skip to content

Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

🕒 Published (v1): 2026-07-11 15:31 UTC · Source: Arxiv · link

Why this paper was selected

Benchmarks 15 foundation model backbones for mammography under domain shift; directly relevant to medical VLM eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks 15 foundation-model backbones as frozen feature extractors for mammography across three clinical tasks (breast density, BI-RADS, cancer status), evaluating OOD generalization on 12 external datasets from 12 countries. Mammography-specific VLMs (MaMA, Mammo-FM) achieve the strongest mean OOD performance, but mammography pretraining alone does not guarantee robustness. DINOv3 remains a surprisingly competitive vision-only baseline, outperforming many mammography-adapted models.

Problem

Prior work has not systematically characterized whether foundation-model representations for mammography generalize under real-world distribution shift. In-distribution (ID) performance is routinely reported, but external OOD generalization across diverse datasets, acquisition protocols, and countries is poorly understood. Heterogeneous clinical label schemes across public datasets further complicate cross-dataset comparison.

Method

A unified frozen-backbone linear-probe protocol is applied: backbone weights are never updated; only a task-specific linear classifier is trained on one source dataset per task (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer). Evaluation uses macro-averaged one-vs-rest AUROC (BI-RADS, density) and binary AUROC (cancer), with 95% CIs from 1000 bootstrap runs. Labels are harmonized across all 15 datasets before evaluation. Images are resized to \(1024 \times 768\); exam-level predictions are obtained by per-class max-pooling of image-level scores. Feature-space inspection is conducted with UMAP on joint EMBED/VinDr embeddings, coloring by dataset, manufacturer, view, density, and BI-RADS.

Key Contributions

  • Harmonized benchmark spanning 15 public mammography datasets from 12 countries/regions, covering three clinically distinct tasks.
  • Systematic evaluation of 15 backbones across five pretraining families: natural-image SSL, general-radiology SSL, mammography-adapted SSL, general medical VLM, and mammography-specific VLM.
  • Demonstration that mammography exposure alone (continued SSL pretraining on a single dataset) does not reliably improve OOD generalization.
  • Per-dataset analysis revealing heterogeneous OOD behavior even for the best-performing models.
  • Feature-space analysis linking embedding geometry (density/view structure) to OOD task performance and pretraining supervision type.

Results

  • Best BI-RADS OOD: Mammo-FM \(0.688 \pm 0.016\); best vision-only: DINOv3 \(0.635 \pm 0.017\) (outperforms all mammography-adapted SSL models).
  • Best density OOD: MaMA \(0.865 \pm 0.006\); DINOv3 \(0.848 \pm 0.006\).
  • Best cancer OOD: MaMA \(0.718 \pm 0.014\), Mammo-FM \(0.714 \pm 0.015\); DINOv3 \(0.677 \pm 0.015\).
  • VLMs outperform vision-only models on average: BI-RADS \(0.624\) vs. \(0.593\); density \(0.850\) vs. \(0.835\); cancer \(0.651\) vs. \(0.601\).
  • Mammography-specific VLMs outperform general medical VLMs on all three tasks.
  • DINOv3-EMBED (continued pretraining on EMBED) decreases BI-RADS (\(0.601\)) and cancer (\(0.626\)) OOD vs. DINOv3 (\(0.635\), \(0.677\)).
  • ID–OOD rank correlation: \(\rho = 0.89\) (BI-RADS), \(\rho = 0.76\) (density), \(\rho = 0.61\) (cancer); mean OOD gaps \(-0.092\), \(-0.092\), \(+0.017\) respectively.
  • UMAP shows MaMA has the clearest density cluster structure; Mammo-FM shows better cross-view alignment, consistent with their task-specific OOD advantages.

Limitations

  • Linear-probe protocol measures representation quality, not best achievable task performance; full fine-tuning is not evaluated.
  • Cancer-status OOD results are noisy due to heterogeneous label definitions across datasets (BI-RADS-derived vs. curated annotations, screening vs. cancer-enriched cohorts).
  • Feature-space interpretations (UMAP) are qualitative.
  • Uncertainty estimation and confidence calibration of backbones across OOD datasets are not evaluated.
  • Mammography-adapted SSL models were pretrained only on EMBED (single source), limiting conclusions about multi-source mammography adaptation.

Relevance to Vision-Language Models

This benchmark directly compares mammography-specific VLMs against vision-only SSL baselines under a controlled OOD protocol, providing evidence that language alignment with clinical text or metadata is a stronger driver of OOD generalization than visual domain exposure alone. The finding that pretraining text source matters—report-based supervision (Mammo-FM) vs. metadata-derived reports (MaMA) yields complementary strengths across tasks—is directly relevant to VLM design choices. The competitive performance of DINOv3 challenges the assumption that modality-specific VLMs automatically dominate, and the feature-space analysis offers a mechanism: VLMs that encode clinical semantics (density, BI-RADS) in their representations transfer those signals more robustly. For researchers tracking VLMs, this is a rigorous case study of how contrastive language-image pretraining interacts with clinical label provenance and acquisition diversity.