A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology¶

🕒 Published (v1): 2026-02-11 08:14 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DermFM-Zero is a dermatology vision-language foundation model trained on 4M+ multimodal data points via masked latent modelling and contrastive learning, achieving zero-shot clinical performance without task-specific fine-tuning. In multinational reader studies with 1,100+ clinicians, it nearly doubled GPs' differential diagnostic accuracy and outperformed board-certified dermatologists in standalone skin cancer assessment. Sparse autoencoders applied to its latent space automatically surface interpretable clinical concepts and enable targeted artifact-bias suppression without retraining.

Problem¶

Existing medical foundation models require task-specific fine-tuning, making them impractical for the open-ended distribution of conditions encountered in real clinical settings. Dermatology-specific vision-language models (MONET, DermLIP) were limited by training scale and generic architectures. Rigorous evaluation of zero-shot models in human-AI collaborative workflows had not been conducted, leaving real-world utility unvalidated.

Method¶

Two-stage training on 4M+ data points: (1) masked latent modelling on 3M unlabeled dermatological images (dermoscopy, clinical photography, mobile) to learn fine-grained morphological features in the vision encoder; (2) bootstrapped contrastive learning aligning visual representations to medical text via PubMedBERT (extended token window) on 1M curated image-text pairs covering 400+ skin conditions. Data sourced from Derm1M, educational corpora, and private MoleMap clinical collections. For interpretability, Sparse Autoencoders (SAEs) disentangle the pretrained vision features into sparse latent neurons, which are auto-named via the text encoder and used to build Concept Bottleneck Models (CBMs) or to suppress artifact-associated neurons at inference.

Key Contributions¶

First zero-shot dermatology VLM validated through large-scale multinational reader studies (n > 1,100 clinicians across primary and specialist care).
State-of-the-art zero-shot performance across 20 benchmarks spanning classification (7 datasets), cross-modal retrieval, and VQA—without task-specific adaptation.
"Discover-then-name" SAE framework for unsupervised clinical concept extraction that outperforms predefined-vocabulary CBMs (MONET paradigm) specifically because DermFM-Zero's representations are sufficiently structured.
Targeted inference-time bias mitigation via concept-neuron suppression (12–38% AUROC gain on artifact-confounded subsets) without retraining.
Demonstrated "skill-leveling" effect: AI-assisted non-experts surpass unassisted experts in both diagnostic accuracy and management appropriateness in a zero-shot setting.

Results¶

Zero-shot classification: HAM-10000 balanced accuracy 0.744 (+23.7% over next-best DermLIP, P<0.001); PAD-UFES-20 0.743 (+23.2%); SNU-134 0.452 (+22.3%); SD-128 0.498 (+20.9%); DAFFODIL-5 (rare diseases) 0.893 (+15.6%). Average zero-shot: DermFM-Zero 73.3% vs. DermLIP 56.1%.
Cross-modal retrieval: Derm1M R@50 I2T 0.601, T2I 0.598 (+32.3% over BiomedCLIP); SkinCap R@50 I2T 0.623, T2I 0.586 (+23.8%/22.6% over MONET). Average R@50: 60.2% vs. MONET 31.9%.
Reader study 1 (primary care, n=30 GPs, 98 conditions): Top-3 accuracy 0.266→0.482 (P=0.004); mean diagnostic score 2.24→3.05 (P=0.006); appropriate management 0.504→0.592 (P=0.018); harmful decisions 0.400→0.341 (P=0.048).
Reader study 2A (standalone vs. 1,090 clinicians): DermFM-Zero 71.7% vs. all clinicians 66.3% (+5.4%, P<0.001); vs. GPs 59.6% (+12.1%); vs. dermatologists 69.4% (+2.3%). Legacy supervised ResNet50 (Ypsono) underperformed clinician average by 9.6%.
Reader study 2B (specialist collaboration, n=34): Accuracy 0.50→0.61 (P<0.001); management appropriateness 0.70→0.73 (P=0.010); AI-assisted non-experts reached 0.59 accuracy vs. unassisted experts 0.55.
Low-resource transfer: 9.6% average improvement over PanDerm with 10% labeled data; outperforms DINOv3-7B (23× larger) across data fractions.
Melanoma prognosis: Hazard Ratio 4.70 (P<0.001); AUCs at 3/5/7 years: 0.930/0.945/0.912.
SAE-CBM: melanoma AUROC 0.939 vs. predefined-concept MONET CBM 0.765 (approaching black-box probe 0.947).
Artifact suppression: AUROC increases of 12–38% across ruler, pen-marking, and hair-occlusion subsets.

Limitations¶

Reader studies are retrospective simulations; no prospective or "silent" deployment validation.
Pretraining covers 400+ of 2,000+ known skin conditions; long-tail conditions remain underrepresented.
Skin-tone-based fairness in human-AI interaction was not specifically investigated despite geographically diverse pretraining data.
Interpretability features (SAE concepts, bias suppression) were not integrated into the collaborative workflow studies; their effect on clinician trust and decision-making is unmeasured.

Relevance to Foundation Models in Medicine¶

DermFM-Zero directly advances the field's core challenge: closing the gap between foundation model technical capability and clinical deployment by demonstrating effective zero-shot generalization without fine-tuning at scale. The SAE-based interpretability framework offers a practically significant path toward explainable, auditable medical AI—moving from predefined concept vocabularies to data-driven concept discovery—which is broadly applicable across medical imaging specialties. The skill-leveling finding (AI-assisted non-experts surpassing unassisted experts in a zero-shot regime) strengthens the evidence base for deploying general-purpose medical FMs in resource-limited settings. The methodology—domain-specific two-stage pretraining producing representations that enable both strong transfer and emergent interpretability—provides a replicable template for other medical imaging domains developing their own foundation models.