Skip to content

A Hierarchical Benchmark of Foundation Models for Dermatology

🕒 Published (v1): 2026-01-18 12:38 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks ten foundation models (general vision, general medical, and dermatology-specific) on hierarchical skin lesion classification using DERM12345 (12,345 dermoscopic images, 40 subclasses). It reveals a "granularity gap": models excelling at binary malignancy detection degrade sharply at fine-grained subtype discrimination, and vice versa. The finding has direct implications for model selection in clinical dermatology AI pipelines.

Problem

Existing dermatology benchmarks reduce complex diagnostic taxonomies to flat binary tasks (e.g., melanoma vs. benign nevi), masking a model's ability to perform fine-grained differential diagnosis across the full clinical hierarchy. No prior work systematically compared foundation model embeddings across all levels of diagnostic granularity using a frozen-feature + lightweight-adapter protocol.

Method

Frozen embeddings are extracted from ten foundation models across three categories: general vision (DINOv2-Base/Giant, DINOv3, CLIP-Base/Large, ResNet-50), general medical (MedSigLip, BiomedCLIP, MedImageInsights), and dermatology-specific (MONET, PanDerm-Base/Large, Derm Foundation). Six lightweight adapter classifiers (KNN, LR, SVM, RF, MLP, XGBoost) are trained on 40-class subclass labels via 5-fold stratified cross-validation with GridSearchCV hyperparameter tuning. Hierarchical evaluation is performed by probability aggregation: fine-grained 40-class softmax outputs are summed over constituent subclasses to yield predictions at 15 Main Classes, 4 Superclasses, 2 Superclasses (melanocytic vs. non-melanocytic), and binary malignancy—without retraining separate heads per level.

Key Contributions

  • Introduces a four-level hierarchical evaluation framework for dermatology foundation models, exposing the "granularity gap" phenomenon
  • Benchmarks ten models spanning three pretraining domains under a standardized frozen-feature protocol on DERM12345 (40 subclasses, 12,345 images)
  • Demonstrates performance inversion: MedImageInsights leads at coarse levels; MedSigLip and dermatology-specific models lead at fine-grained 40-class classification
  • Shows that MLP adapters consistently outperform linear probes and distance-based methods for fine-grained tasks, and that "best-of-suite" adapter selection is necessary for fair comparison
  • Identifies the "Blob Problem" via t-SNE: melanocytic nevi subclasses (banal vs. dysplastic) remain unseparated in all embedding spaces without supervised adaptation

Results

  • Binary malignancy (2-class): MedImageInsights best at 97.52% weighted F1; MedSigLip second at 96.43%
  • 2-class Superclass (melanocytic vs. non-melanocytic): MedImageInsights 95.39%; MedSigLip 93.85%
  • 4-class Superclass: MedImageInsights 93.45%; MedSigLip 91.66%
  • 15-class Main Class: MedImageInsights 62.40%; MedSigLip 62.29%
  • 40-class Subclass (fine-grained): MedSigLip 69.79% > Derm Foundation 69.50% > MONET 69.31% > DINOv2-Giant 68.00% > MedImageInsights 65.50%
  • ResNet-50 baseline: 58.82% at subclass level—>10 pp behind leading foundation models
  • PanDerm-Large collapses to 36.65% (below ResNet-50), suggesting a defective public checkpoint
  • MLP adapter on MedSigLip (69.8%) vs. SVM (46.9%): large adapter-choice sensitivity
  • MedImageInsights confusion matrix: 27.93% error rate misclassifying Dysplastic Compound Nevi as Banal Compound Nevi

Limitations

  • DERM12345 sourced exclusively from Türkiye; skin phototype distribution may not generalize globally
  • Evaluation restricted to dermoscopic images; clinical macroscopic photography not assessed
  • Hierarchical aggregation from 40-class predictions is one specific evaluation strategy; direct multi-level training may yield different relative rankings
  • PanDerm-Large anomaly not investigated beyond suspecting a defective checkpoint
  • No zero-shot or few-shot evaluation; all adapters require supervised training data

Relevance to Foundation Models in Medicine

This benchmark directly addresses the open question of whether general medical foundation models or domain-specific ones are better suited for clinical dermatology, finding that the answer is task-granularity-dependent—a key practical insight for deploying foundation models in real workflows. The "granularity gap" identified here mirrors concerns in other medical imaging domains (pathology, radiology) where high-level semantic alignment from image-text pretraining may come at the cost of fine-grained visual discrimination. The frozen-embedding + lightweight-adapter protocol is a standard evaluation paradigm for medical foundation models, and this paper's four-level hierarchical framework provides a reusable template. The finding that MedSigLip (a general medical model) matches or exceeds dermatology-specific models at the subclass level challenges the assumption that domain-specific pretraining always wins, with implications for resource allocation in foundation model development for medicine.