Skip to content

BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis

🕒 Published (v1): 2025-05-14 12:25 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BioVFM-21M introduces a 21-million-image multi-modal biomedical dataset and uses it to systematically study SSL scaling laws for medical vision foundation models across model size, data size, modality diversity, and pretraining algorithm. The resulting BioVFM model (630M parameters, DINOv2-based) outperforms prior SOTA across 12 MedMNIST benchmarks. Crucially, scaling benefits are highly task-dependent and data diversity matters more than raw data volume.

Problem

Scaling laws are well-characterized for general-domain vision/language models, but it is unknown whether and how they transfer to medical imaging, which differs substantially from natural images. Existing medical FM datasets are modality- or organ-specific, limiting diversity and preventing rigorous cross-scale empirical study. Without this understanding, practitioners cannot make principled decisions about when additional compute or data is worth the cost.

Method

The authors curate BioVFM-21M: 21M 2D images (after removing padding slices and aspect-ratio outliers from an initial 30M) drawn from 43 public sources, spanning 10 modalities (CT, MRI, X-ray, dermoscopy, ultrasound, endoscopy, fundus, microscopy, nuclear medicine, OCTA) and 30 anatomical regions; all benchmark-related sources are excluded to prevent leakage. Two SSL algorithms are compared: MAE (BioVFM-M) and DINOv2 (BioVFM-D), each trained at model sizes from 5M to 303M parameters. Data/modality scaling is studied by comparing BioVFM-US (0.4M ultrasound-only), BioVFM-0.2M (0.2M, all 10 modalities), and full BioVFM-21M. Evaluation uses linear-probing AUC on 12 MedMNIST classification benchmarks. To quantify scaling efficiency, a power-law curve y = e^b · x^a is fit per task, and Pearson correlation is computed between scaling slope a and four task-level metrics: sample count, Davies-Bouldin Index (DBI), class count, and gzip compressibility.

Key Contributions

  • BioVFM-21M dataset: 21M biomedical images across 10 modalities and 30 anatomies from 43 public sources, the largest open multi-modal biomedical image dataset to date.
  • Systematic scaling study: Empirical sweep over model size (5M–303M), SSL algorithm (MAE vs. DINOv2), data size, and modality breadth with power-law fitting per task.
  • Correlation analysis: Identifies dataset-level metrics (DBI, sample count, gzip compressibility) as predictors of scaling benefit via Pearson correlation with scaling slopes.
  • BioVFM model: 630M-parameter DINOv2-pretrained model setting new SOTA across 12 MedMNIST benchmarks.

Results

  • BioVFM (630M, DINOv2, 21M images) vs. best prior SOTA (BiomedCLIP): +3.32% MCC, +2.81% BA, +2.14% F1, +0.94% AUC, averaged over 12 benchmarks.
  • Model scaling: BioVFM-D improves mean AUC by 2.1% and BioVFM-M by 2.3% going from 5M to 303M parameters; however, per-task scaling slopes vary >40-fold (e.g., ChestMNIST +4.65% AUC vs. RetinaMNIST plateau beyond 21.7M params).
  • DINOv2 vs. MAE: DINOv2 outperforms MAE by 1.7% average AUC across all sizes despite MAE having a steeper scaling slope; BioVFM-D (85M) beats BiomedCLIP on 7/12 tasks.
  • Modality diversity scaling: adding modalities (BioVFM-US → BioVFM-0.2M) yields +0.97% mean AUC; ultrasound-specific performance improves from 91.87% to 92.75% despite inclusion of other modalities.
  • Data volume scaling: 100× more data (0.2M → 21M) yields only +0.2% mean AUC gain, indicating strong diminishing returns.
  • Correlation analysis: sample count (positive) and gzip compressibility (negative) show strongest Pearson correlation with scaling slope; DBI also positively correlated.

Limitations

  • Evaluation is restricted to 12 MedMNIST classification tasks; segmentation, detection, and generation are not assessed, limiting generalizability of scaling conclusions.
  • Linear-probing only; full fine-tuning and few-shot transfer behaviors may differ.
  • 3D volumetric context is discarded (3D images sliced to 2D axial/sagittal/coronal), potentially losing spatial relationships critical for volumetric tasks.
  • Power-law fitting is applied to a relatively narrow parameter range (5M–303M), and extrapolation to larger scales (e.g., 1B+) is not validated.
  • Dataset construction excludes labeled data and relies only on SSL; the interaction between pretraining scale and label-efficient fine-tuning is unexplored.

Relevance to Foundation Models in Medicine

BioVFM-21M provides the field with both a large-scale open pretraining resource and a concrete empirical account of when scaling actually helps in the medical domain—directly relevant to anyone deciding how to invest compute for medical FMs. The finding that data diversity dominates over data volume challenges the "just add more data" assumption and suggests dataset curation strategy is at least as important as scale. The correlation of task-level DBI and gzip compressibility with scaling efficiency offers actionable heuristics for predicting which clinical tasks will benefit from larger models. This work also establishes a reproducible SSL-only baseline that isolates scaling effects from supervision signals, complementing concurrent vision-language scaling studies like RAD-DINO.