PEFT-MedSAM: Efficient Fine-Tuning of Medical Foundation Models for Explainable Skin Lesion Segmentation

🕒 Published (v1): 2026-06-17 05:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PEFT-MedSAM adapts MedSAM (a medical foundation model) to dermoscopic skin lesion segmentation by freezing the ViT-B image encoder and prompt encoder and fine-tuning only the 4M-parameter mask decoder (4.3% of total parameters). On ISIC 2018, it achieves Dice = 0.9411 / IoU = 0.8918, outperforming both a fully trained U-Net baseline and zero-shot MedSAM inference. Grad-CAM with a Pointing Game metric provides quantitative explainability evidence, a prerequisite for clinical deployment.

Problem

Full fine-tuning of MedSAM's 93M parameters is computationally prohibitive for most institutions and risks catastrophic forgetting of broadly learned medical visual representations. Existing dermoscopic segmentation methods train task-specific architectures from scratch, ignoring pre-trained representations, and the literature lacks quantitative explainability evaluation for segmentation—only qualitative visualizations or classification-focused XAI exist.

Method

PEFT-MedSAM freezes MedSAM's image encoder (ViT-B, ~88M parameters, 1024×1024 input) and prompt encoder (~6K parameters); only the lightweight two-layer transformer mask decoder (~4M parameters) is updated during fine-tuning on ISIC 2018 (2,075 training images). Bounding-box prompts are generated automatically from ground-truth masks with 5-pixel padding. The training loss combines Focal loss and Dice loss:

\[\mathcal{L} = \mathcal{L}_{\text{Focal}} + \mathcal{L}_{\text{Dice}}\]

with \(\alpha=0.25\), \(\gamma=2.0\) in Focal loss to address lesion/background class imbalance. Optimization uses Adam with lr \(= 1\times10^{-5}\), ReduceLROnPlateau scheduling, and geometric+color augmentation (with bounding-box coordinate recomputation to maintain spatial consistency). Explainability is evaluated by applying Grad-CAM to the U-Net CNN baseline and scoring it with the Pointing Game metric:

\[\text{PGA} = \frac{N_{\text{hit}}}{N_{\text{total}}}\]

where \(N_{\text{hit}}\) counts images whose Grad-CAM maximum activation falls within the ground-truth lesion mask.

Key Contributions

• PEFT strategy fine-tuning only 4.3% of MedSAM parameters, achieving Dice = 0.9411 on ISIC 2018.
• Systematic three-way comparison: task-specific CNN (U-Net), zero-shot foundation model (MedSAM), and PEFT foundation model (PEFT-MedSAM), with Wilcoxon signed-rank statistical validation (\(p < 0.0001\)).
• Quantitative explainability via Grad-CAM + Pointing Game on CNN baseline, reporting 98.27% accuracy on 519 validation images.
• Cross-dataset generalization: external validation on PH2 (200 unseen images, different imaging protocol), Dice = 0.9467 ± 0.0310.

Results

• ISIC 2018: PEFT-MedSAM Dice = 0.9411, IoU = 0.8918, std = ±0.0480, 95% CI [0.9364, 0.9447].
• U-Net baseline (ISIC 2018): Dice = 0.8715, IoU = 0.7951, std = ±0.1481. PEFT-MedSAM improvement: +7.99% Dice.
• MedSAM zero-shot (ISIC 2018): Dice = 0.8997, IoU = 0.8248. PEFT-MedSAM improvement: +4.60% Dice.
• PH2 external validation: PEFT-MedSAM Dice = 0.9467, IoU = 0.9003 (vs. U-Net 0.8961 / MedSAM zero-shot 0.9139).
• Explainability (Pointing Game, CNN baseline): PGA = 98.27% on 519 ISIC 2018 validation images.
• Wilcoxon \(p < 0.0001\) confirming statistical significance of PEFT-MedSAM over baselines.

Limitations

• Bounding-box prompts are derived from ground-truth masks; clinical deployment requires manual or automated prompt generation without ground-truth access.
• Explainability (Grad-CAM/Pointing Game) is applied only to the U-Net CNN baseline, not to the transformer-based PEFT-MedSAM itself—gradient-based saliency methods are ill-suited for ViT architectures.
• Training capped at 39 epochs due to a computational time limit rather than full convergence.
• Evaluation limited to two dermoscopic datasets; no multi-class or multi-modality validation beyond skin lesions.
• ISIC 2018 training set is relatively small (2,075 images), potentially limiting generalization to rare lesion types not represented in either dataset.

Relevance to Foundation Models in Medicine

This paper is a direct study of how to efficiently adapt a large medical foundation model (MedSAM) to a specialized clinical task with minimal trainable parameters, demonstrating that decoder-only fine-tuning preserves generalizable encoder representations while achieving state-of-the-art segmentation—a practically important finding for resource-constrained clinical AI deployment. The stark reduction in standard deviation (0.0480 vs. 0.1481 for U-Net) illustrates a reliability benefit of foundation model pre-training beyond average accuracy, which is underreported in the literature. The paper also surfaces a key unresolved gap: quantitative explainability methods for transformer-based segmentation models remain lacking, which is increasingly relevant as regulatory bodies require interpretability alongside performance for clinical AI tools.