Skip to content

Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomics

🕒 Published (v1): 2025-09-03 15:48 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically benchmarks generalist vision foundation models (DINOv2, DINOv3) against retina-specialist models (RETFound-MAE, RETFound-DINOv2) across ten ocular disease detection and oculomics tasks. RETFound-DINOv2 consistently outperforms generalist models in both predictive performance and data efficiency, confirming that domain-specific pre-training remains valuable. However, DINOv3-ViT-large closes the gap substantially, suggesting that generalist scaling is a viable path toward medical-grade representations.

Problem

As generalist vision foundation models scale to billions of training images (DINOv2: 142M, DINOv3: 1.69B), it is unclear whether domain-specific pre-training on medical images still provides a meaningful advantage, or whether sufficient scale and improved training strategies (e.g., Gram Anchoring in DINOv3) can substitute for it in clinical retinal image tasks.

Method

Four model families (DINOv2-ViT-{S,B,L,G}; DINOv3-ViT-{S,S+,B,L,H+}; RETFound-MAE-ViT-L; RETFound-DINOv2-ViT-L) are evaluated on ten downstream tasks: diabetic retinopathy grading (APTOS-2019, IDRiD, MESSIDOR2), glaucoma detection (Papila, Glaucoma Fundus), multi-disease classification (Kaggle Retina), and 3-year systemic disease prediction (MI, stroke, heart failure, Parkinson's) using the AlzEye cohort for fine-tuning and UK Biobank for external validation. Two adaptation strategies are compared—full fine-tuning vs. linear probing—with data efficiency evaluated at 10/30/50/70/100% training subsets. Adaptation efficiency (GPU memory, runtime on A100) and feature discriminability (cosine self-similarity heatmaps on 250 UK Biobank images) are also characterized.

Key Contributions

  • Comprehensive head-to-head benchmark of generalist (DINOv2, DINOv3) vs. specialist (RETFound) ViT foundation models on 10 retinal clinical tasks with internal and external validation.
  • Demonstration that DINOv3-ViT-large outperforms DINOv2-ViT-giant (avg AUROC 0.816 vs. 0.800) despite being ~3.5Ă— smaller, attributing the gain to larger pre-training data and Gram Anchoring.
  • Evidence that domain scaling laws from natural images do not transfer directly: DINOv3-ViT-huge+ underperforms ViT-large on retinal tasks.
  • Quantification of RETFound-DINOv2's feature discriminability advantage: lowest pairwise cosine self-similarity (0.798) among the four top models, correlating with superior downstream separation.
  • Adaptation efficiency analysis showing ViT-L–backed models (DINOv3-ViT-L, RETFound-DINOv2, RETFound-MAE) are ~3.5Ă— cheaper in GPU memory than DINOv2-ViT-giant for fine-tuning.

Results

  • Overall fine-tuning AUROC (avg over 10 tasks): RETFound-DINOv2 0.830 > DINOv3-ViT-L 0.816 > RETFound-MAE 0.809 > DINOv2-ViT-G 0.800; RETFound-DINOv2 significantly outperforms DINOv3-ViT-L (Wilcoxon p<0.05).
  • Average rank (fine-tuning): RETFound-DINOv2 1.7, DINOv3-ViT-L 2.3, RETFound-MAE 2.8, DINOv2-ViT-G 3.2.
  • Linear probing AUROC: RETFound-DINOv2 0.787 (best); fine-tuning outperforms linear probing for all models (range 0.800–0.830 vs. 0.761–0.787).
  • External validation (UK Biobank, systemic disease): RETFound-DINOv2 avg AUROC 0.599 > RETFound-MAE 0.571 > DINOv2-ViT-G 0.560 > DINOv3-ViT-L 0.549.
  • External validation (ocular, F1): RETFound-DINOv2 0.402 > DINOv3-ViT-L 0.381 > RETFound-MAE 0.367 > DINOv2-ViT-G 0.365.
  • Data efficiency (10% training data, avg AUROC): DINOv3-ViT-L and RETFound-DINOv2 tied at avg rank 2.2; RETFound-DINOv2 retains 92.6% of full-data AUROC for heart failure at 10% training.
  • Feature self-similarity: RETFound-DINOv2 0.798, DINOv2-ViT-G 0.897, DINOv3-ViT-L 0.915, RETFound-MAE 0.975.

Limitations

  • Evaluation restricted to classification tasks (disease detection/prediction); segmentation and regression benchmarks are absent.
  • DINOv3-ViT-7B excluded due to compute requirements, leaving the largest generalist model uncharacterized.
  • No medical foundation models built on DINOv3 backbone are included (DINOv3 was recently released).
  • Analysis covers only retinal fundus images; generalizability to other ophthalmic modalities (OCT, SLO) or other medical imaging domains is untested.
  • AlzEye data access is restricted, limiting reproducibility for systemic disease prediction experiments.
  • Reasons for DINOv3-ViT-small+ and ViT-huge+ underperformance on retinal data are not mechanistically explained.
  • Multimodal and vision-language models (PaliGemma, Qwen) are not evaluated.

Relevance to Foundation Models in Medicine

This paper directly addresses the core tension in medical foundation model development: whether domain-specific pre-training remains necessary as generalist models scale. The finding that RETFound-DINOv2—which bootstraps DINOv2-ViT-L with 904K retinal images—consistently outperforms billion-parameter generalist models reinforces the value of continued domain-adaptive pre-training pipelines rather than relying solely on general-purpose scaling. The result that DINOv3 substantially narrows the gap using better training strategies (not just more parameters) has direct implications for how the field should invest compute: improved SSL objectives and curated data may matter more than raw model size. The feature discriminability analysis (self-similarity as a proxy for representation quality) offers a transferable diagnostic for evaluating any candidate medical foundation model backbone.