Skip to content

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

🕒 Published (v1): 2026-03-14 00:06 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MASS (MAsk-guided Self-Supervised learning) introduces in-context segmentation as a pretext task for 3D medical imaging, using automatically generated class-agnostic masks from SAM2 to learn semantically rich representations without any expert annotations. Trained on 5K multi-modal CT/MRI/PET volumes, MASS achieves few-shot segmentation competitive with full supervision and frozen-encoder classification that exceeds fully supervised training from scratch on unseen pathologies.

Problem

Existing SSL methods for 3D medical imaging rely on low-level reconstruction (MAE, Model Genesis) or contrastive objectives (SimCLR) that fail to encode the anatomical semantics—spatial localization, shape, appearance, and inter-structure relationships—required for downstream clinical tasks. Supervised pretraining is constrained by predefined taxonomies and prohibitive annotation costs, preventing the development of general-purpose 3D medical foundation models.

Method

MASS formulates pretraining as in-context segmentation (ICS): given an image-mask reference pair (x_s, y_s), predict the same anatomical region in an augmented query view (x_q, y_q) of the same volume. Masks are generated annotation-free via SAM2 in auto mode—2D slices are sampled, SAM2 segments them, and predictions are propagated through the 3D volume—yielding hundreds to thousands of class-agnostic masks per volume. The architecture (following Iris) comprises image encoder E_θ, task encoding module T_φ, and decoder D_ψ. The task embedding T = T_φ(E_θ(x_s), y_s) conditions the decoder to produce ŷ_q. Loss is Dice + BCE. Semantic identity is learned implicitly: appearance and spatial augmentations break all low-level shortcuts, forcing the encoder to represent what remains invariant—anatomical structure identity. No semantic labels are ever used.

Key Contributions

  • Novel SSL pretext task for 3D medical imaging: in-context segmentation with auto-generated class-agnostic masks (SAM2), eliminating expert annotation entirely.
  • Annotation-free scalability from 20-scan single-dataset pretraining to 5K-volume multi-modal (CT/MRI/PET) pretraining.
  • Few-shot segmentation: MASS-FT with 20–40% labeled data matches full supervision; exceeds best SSL baselines by >20 Dice points in 1-shot settings.
  • Training-free in-context inference (MASS-IC): surpasses all finetuned SSL baselines in 1-shot segmentation on anatomical structures without parameter updates.
  • Frozen-encoder classification on entirely unseen pathologies (intracranial hemorrhage, trauma) matches or exceeds full supervised training from scratch at low data regimes.

Results

Few-shot segmentation (single-dataset, small-scale): - BCV 1-shot: MASS-FT 68.8% vs. best baseline (SimCLR) 44.9% (+23.9 Dice); MASS-IC 65.5%, surpassing all finetuned baselines without finetuning. - AMOS MR 1-shot: MASS-FT 65.9% vs. MAE best 36.2% (+29.7 Dice). - KiTS tumor 30-shot: MASS-FT 64.3% vs. MAE best 42.2% (+22.1 Dice); full supervision 81.7%.

Few-shot segmentation (large-scale multi-modal pretraining on 5K volumes): - BCV 1-shot: MASS-FT 70.2%, best SSL baseline (AnatoMix) 53.1% (+17.1); Iris (supervised) 83.2%. - AMOS MR 1-shot: MASS-FT 74.3%, best SSL (Merlin) 37.9% (+36.4). - Pelvic 1-shot: MASS-FT 92.8% vs. supervised Iris 86.9%, SuPreM 85.4%. - KiTS tumor 30-shot: MASS-FT 70.0%, best SSL (Merlin) 62.7% (+7.3).

Frozen-encoder classification (unseen pathologies): - RSNA ICH 5% data (761 samples): MASS 75.4% AUC vs. full training from scratch 72.8%; next best SSL (OM-SimCLR) 58.9%. - Liver trauma 30%: MASS 86.7% AUC vs. full training 74.4%. - Kidney trauma 30%: MASS 82.9% vs. full training 75.0%. - Spleen trauma 30%: MASS 85.5% vs. full training 82.1%.

Mask quality analysis: - SAM2-generated masks have only 15.2% average Dice overlap with ground truth on BCV, yet MASS achieves 65.5% 1-shot ICS, demonstrating robustness to weak supervision.

Limitations

  • Expert-annotated masks are deliberately excluded from pretraining; synergy between auto-generated and expert masks is unexplored.
  • MASS-IC fails on highly variable pathological structures (e.g., KiTS tumors: 2.7% 1-shot Dice), requiring finetuning to recover.
  • No vision-language alignment: representations are not connected to radiology reports or text, limiting utility for report generation and VL reasoning.
  • Supervised baselines (Iris, SuPreM) outperform MASS by 10–15 Dice points when evaluation targets align with their pretraining taxonomies.
  • Scaling experiments use pretrained baseline weights rather than retraining on the same corpus, making direct comparison partially confounded.

Relevance to Foundation Models in Medicine

MASS directly advances the core challenge in 3D medical foundation models: learning transferable, semantically grounded representations without annotation bottlenecks. By treating in-context segmentation as the pretext task and leveraging off-the-shelf class-agnostic segmenters (SAM2) for weak supervision, it demonstrates a scalable path to annotation-free pretraining that rivals supervised approaches on out-of-distribution tasks—a key requirement for clinical generalization. The frozen-encoder classification results are especially significant, showing that dense segmentation pretraining yields broadly transferable features for image-level pathology tasks, a property that contrastive and reconstruction SSL methods lack. This positions MASS as a complementary approach to vision-language medical foundation models, with the missing text alignment being the natural integration point for future work.