Skip to content

Uncovering Modality Discrepancy and Generalization Illusion for General-Purpose 3D Medical Segmentation

🕒 Published (v1): 2026-02-07 17:54 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current 3D medical segmentation foundation models (SAM-Med3D-turbo, SegVol, SAT-Pro, VISTA3D, nnInteractive) are evaluated on a rigorously controlled paired PET/CT and PET/MRI benchmark and found to catastrophically fail on functional (PET) imaging despite strong reported benchmarks on structural modalities. The UMD dataset isolates imaging modality as the sole variable via intra-subject paired acquisition, exposing a systemic structural bias. All five tested foundation models fall far below task-specific nnU-Net baselines trained on only 10 cases per modality.

Problem

Existing evaluations of 3D medical segmentation foundation models conflate modality differences with task complexity: CT benchmarks use abdominal organs while MRI benchmarks use cardiac or neurological structures, making it impossible to isolate modality as the independent variable. Additionally, training data across all evaluated models is overwhelmingly dominated by structural imaging (CT/MRI), with PET constituting a negligible fraction, creating a structural bias that is masked by curated test splits. No prior benchmark used intra-subject paired structural/functional scans to isolate modality discrepancy.

Method

The authors curate the UMD (Universal Modality Discrepancy) dataset: 490 whole-body PET/CT and 464 whole-body PET/MRI scans acquired in the same diagnostic session (PET/CT: Siemens Biograph 64; PET/MRI: 3.0T Siemens Biograph mMR), with voxel-level annotations for 13 organs (liver, kidneys, brain, heart, spleen, aorta, lung, colon, bladder, pancreas, esophagus, stomach). Intra-subject pairing ensures identical anatomy, scale, and orientation across modalities, making modality the sole experimental variable. Five state-of-the-art 3D foundation models are evaluated zero-shot using their official weights and standardized prompting protocols: semantic/text-guided (VISTA3D, SAT-Pro) and point-based (SAM-Med3D-turbo, nnInteractive, SegVol). A 10-case-per-modality nnU-Net is trained as a task-specific reference.

Key Contributions

  • UMD benchmark: 954 total whole-body scans (PET/CT + PET/MRI) with 13-organ annotations enabling controlled modality comparison; newly collected to preclude data leakage.
  • Intra-subject controlled evaluation design that isolates modality as the primary independent variable, separating it from anatomical task complexity.
  • Empirical exposure of the generalization illusion: documented gap between literature-reported benchmarks and actual zero-shot performance on unseen distributions.
  • Systematic characterization of failure modes across semantic-guided vs. point-based prompting strategies on functional imaging.

Results

  • VISTA3D (semantic-guided): DSC 0.919 on CT liver (sole strong result); near-zero across all other organs and both PET modalities (avg CT: 0.071, avg PET/CT: 0.005, avg PET/MRI: 0.006).
  • SAT-Pro (text-guided): Total collapse on UMD—avg DSC ~0.018 (CT), ~0.003 (PET/CT), ~0.006 (MRI), ~0.000 (PET/MRI).
  • SAM-Med3D-turbo (point-based): Best overall but still limited—avg DSC 0.404 (CT), 0.410 (PET/CT), 0.433 (MRI), 0.397 (PET/MRI); reported benchmark was 0.790.
  • nnInteractive (point-based): avg DSC 0.481 (CT), 0.342 (PET/CT), 0.511 (MRI), 0.217 (PET/MRI); reported benchmark ~0.55.
  • SegVol: avg DSC 0.107 (CT), 0.075 (PET/CT), 0.134 (MRI), 0.148 (PET/MRI); reported benchmark 0.793.
  • nnU-Net (10-shot reference): avg DSC 0.686 (CT), 0.605 (PET/CT), 0.652 (MRI), 0.560 (PET/MRI)—outperforms all foundation models by large margins.
  • Exception: bladder shows localized PET recovery in point-based models due to intense radiotracer accumulation.

Limitations

  • Evaluation restricted to whole-body organs; pathological targets (tumors, lesions) not included.
  • Only Dice Similarity Coefficient (DSC) used; no Hausdorff distance or surface metrics.
  • Limited prompt types evaluated per model (single-point or text/class-ID only; no multi-point, bounding-box, or interactive refinement).
  • Dataset is single-institution (Shanghai), which may limit acquisition-parameter diversity.
  • nnU-Net reference uses only 10 training cases—not a full supervised upper bound.

Relevance to Foundation Models in Medicine

This paper directly challenges the generalization claims of the current generation of 3D medical foundation models by demonstrating that structural imaging performance does not transfer to functional imaging, a critical clinical modality for oncology and metabolic disease. The controlled benchmark design is methodologically important: it provides a template for rigorous modality-stratified evaluation that the field currently lacks. For researchers tracking medical foundation models, the finding that even best-in-class models are outperformed by a 10-case task-specific nnU-Net on PET underscores that scaling on structural data alone is insufficient, and motivates multi-modal pre-training pipelines that include functional imaging. The UMD dataset and analysis establish a public testbed for future modality-agnostic model development.