From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images¶

🕒 Published (v1): 2026-01-25 18:13 UTC · Source: Arxiv · Venue: ISBI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SC-SAM introduces a bidirectional co-training framework pairing U-Net (specialist) with SAM (generalist) for semi-supervised medical image segmentation. U-Net generates point prompts and pseudo-labels to guide PEFT-adapted SAM, while SAM's refined masks regularize U-Net training. The method outperforms MedSAM, SAM-Med2D, and other semi-supervised SAM variants on prostate MRI and polyp benchmarks with only 5–10% labeled data.

Problem¶

SAM's PEFT-based adaptation to medical images relies almost exclusively on labeled data, leaving abundant unlabeled medical images unexploited. Dual-SAM approaches suffer from prediction coupling (both branches converging to the same local minima under domain shift), while specialist-only prompting methods (SP-SAM) use conventional networks solely for prompt generation, not for pseudo-label-driven unsupervised learning.

Method¶

SC-SAM implements a bidirectional co-training loop between a U-Net specialist and PEFT-adapted SAM:

Specialist → Generalist: U-Net is trained on both labeled and unlabeled data via standard semi-supervised techniques. Its predictions are used to sample foreground/background point prompts (5+5 per sample) that guide SAM's PEFT fine-tuning on labeled images. U-Net pseudo-labels from unlabeled images are also used as supervision targets for SAM.
Generalist → Specialist: SAM's refined masks serve as pseudo-labels to regularize U-Net on unlabeled data.
Sigmoid ramp-up: To prevent noisy early-stage U-Net predictions from corrupting SAM, the U-Net→SAM unsupervised loss term is scaled by a sigmoid warm-up function ω(t) = exp(-(1 - t/Tmax)²) that grows from ~0 to 1 over Tmax iterations. The total loss is Ltotal = Lsup + Lunsup where Lunsup contains both directions, with the U-Net→SAM direction gated by ω(t).

SAM uses learnable Adapter layers in each Transformer encoder block; U-Net backbone is standard UNet. Batch size 24 (12 labeled + 12 unlabeled); Adam for SAM (lr=1e-4), SGD (momentum 0.9) for U-Net.

Key Contributions¶

Bidirectional specialist-generalist co-training loop enabling SAM to exploit unlabeled medical data through U-Net-derived point prompts and pseudo-labels
Sigmoid ramp-up schedule that stabilizes early training by gradually introducing noisy U-Net→SAM supervision signals
Empirical demonstration that a conventional CNN specialist can correct SAM's overconfident domain-shifted predictions more effectively than dual-SAM architectures
State-of-the-art results on prostate MRI (PROMISE12) and polyp segmentation (5 test sets) under 5% and 10% label regimes

Results¶

PROMISE12 (prostate MRI): - 5% labels: SC-SAM Dice 83.64, IoU 73.87, HD95 3.98, ASD 3.79 vs. KnowSAM (Dice 78.49), CPC-SAM (73.73), MedSAM (63.00) - 10% labels: SC-SAM Dice 83.35, IoU 73.54 vs. KnowSAM (82.93), SAM (79.66), MedSAM (74.75)

Colon (5% labels, cross-dataset generalization): - CVC-300: SC-SAM Dice 88.72 vs. KnowSAM 83.75, SAM 82.02, MedSAM 17.56 - CVC-ClinicDB: 79.54 vs. KnowSAM 78.86, SAM 74.27 - ETIS-Larib: 56.91 vs. KnowSAM 56.72, SAM 51.09 - Kvasir: 84.79 vs. KnowSAM 84.78

Ablation (5% PROMISE12): - Without sigmoid ramp-up: Dice drops from 83.64 to 36.37 (47-point collapse) - UNet++ backbone: 80.14 Dice; ResUNet++: 78.20; SwinUNet: 70.27 — standard UNet best

Limitations¶

Evaluated only on 2D slices from 3D MRI volumes (PROMISE12), not native 3D volumetric segmentation
Limited to two benchmarks (prostate MRI and polyp); generalizability to other modalities (CT, pathology, X-ray) not demonstrated
Requires two separate models (U-Net + PEFT SAM) in training, increasing memory and compute overhead compared to single-model approaches
SwinUNet underperforms as the specialist backbone due to data-hungry ViT behavior in low-label regimes — framework assumes CNN-based specialist
Experiments run on a single RTX 3090 Ti; scalability to larger datasets not assessed

Relevance to Foundation Models in Medicine¶

SC-SAM directly addresses a core challenge in medical foundation model adaptation: how to leverage SAM's strong priors when labels are scarce and domain shift is severe. By establishing that conventional specialist networks (U-Net) can serve as effective teachers for SAM rather than merely prompt generators, this work opens a complementary pathway to purely foundation-model-centric approaches like MedSAM and SAM-Med2D. The result that vanilla SAM fine-tuned on-domain outperforms MedSAM and SAM-Med2D in low-label settings challenges the assumption that medical pretraining alone is sufficient, suggesting that label-efficient adaptation strategies may matter more than domain-specific pretraining at small data scales. The bidirectional co-training paradigm is broadly applicable to any foundation model that supports PEFT but lacks native semi-supervised learning capabilities.