Skip to content

A Continual Learning-driven Model for Accurate and Generalizable Segmentation of Clinically Comprehensive and Fine-grained Whole-body Anatomies in CT

🕒 Published (v1): 2025-03-16 23:55 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CL-Net is a continual learning-driven unified CT segmentation model that accurately segments 235 fine-grained whole-body anatomies (organs, lymph node stations, lesions) by combining a frozen general encoder with sequentially trained, Lottery Ticket Hypothesis-pruned organ-wise decoders. It resolves the partial-labeling problem across 36 heterogeneous datasets without catastrophic forgetting. CL-Net outperforms both an ensemble of 36 specialist nnUNets and current SAM-style medical foundation models while using only 55.5 MB of parameters.

Problem

No single fully-annotated CT dataset covering all clinically relevant anatomies exists due to annotation cost and expertise requirements. Prior multi-dataset approaches either suffer catastrophic forgetting, face optimization infeasibility at scale (MultiTalent requires >1,000 GPU hours on just five datasets), or produce inferior segmentation relative to dataset-specific specialists. SAM-style medical foundation models underperform nnUNet specialists substantially on 3D CT due to architectural mismatch (2D Transformer, optimization difficulty) and lack of local boundary discrimination.

Method

CL-Net is an architectural continual learning framework with three components:

  1. General Encoder (GE): Trained on one or more initial datasets (e.g., TotalSegmentator V2), then frozen permanently. The frozen encoder provides transferable 3D CT features without re-training, enabling zero-forgetting expansion.

  2. Organ-wise decoders: One lightweight decoder per organ or organ group (e.g., rib instances). In universal segmentation mode, all datasets are simultaneously available and decoders train in parallel. In continual segmentation mode, datasets arrive sequentially; new decoders are added for unseen anatomies and existing decoders are fine-tuned via Exponential Moving Average (EMA) on new data.

  3. LTH-based decoder pruning: Each decoder undergoes Lottery Ticket Hypothesis pruning, retaining ~1–3% of its full-size parameters (mean pruning rate 97.5%). This keeps the total model at 55.5 MB regardless of how many anatomies are added.

  4. Body-part-guided merging module: Uses Body Part Regression (BPR) scores to spatially constrain and merge outputs from all decoders, suppressing distal false positives.

Feature Level Supporting (FL-Supporting) injects features from adjacent organ decoders into context-dependent structures such as lymph node stations.

Training used 13,952 CT scans from 20 public and 16 private datasets across head & neck, chest, abdomen, and pelvis, with varied vendors, contrast phases, and pathology types.

Key Contributions

  • Single unified model segmenting 235 whole-body anatomies (193 organs, 33 lymph node stations, 9 lesions), including 73 fine-grained structures not available in any public dataset.
  • Continual learning framework with zero catastrophic forgetting: frozen encoder + organ-wise EMA decoder fine-tuning enables sequential dataset integration without prior data re-access.
  • LTH-based decoder pruning achieving 97.5% mean pruning rate, compressing 1,612 MB (unpruned) to 55.5 MB with negligible accuracy loss.
  • Supports both universal segmentation (simultaneous data access) and continual segmentation (sequential, single-dataset-at-a-time) configurations within the same architecture.
  • Demonstrated superiority over SAM-style medical foundation models (VISTA3D, SAT-Pro, BiomedParse) and the multi-dataset nnUNet extension MultiTalent by large margins.

Results

  • vs. 36-nnUNet ensemble (internal, 235 anatomies): CL-NetU36: 86.1% DSC / 1.17mm ASD vs. nnUNetE36: 83.9% DSC / 1.32mm ASD; outperforms in 206/235 anatomies; model size 55.5 MB vs. 1,126.8 MB ensemble.
  • Continual mode (CL-NetC36): 85.7% DSC, only 0.4% below simultaneous training, confirming near-zero forgetting.
  • vs. MultiTalent (5-dataset setting): CL-NetU5: 92.9% DSC vs. MultiTalent: 90.2% DSC (+2.7% absolute); ASD 0.77mm vs. 1.13mm (−31.9% relative).
  • vs. SAT-Pro (5-dataset setting): CL-NetU5 +9.9% DSC overall; +24.1% DSC on StructSeg19 head & neck.
  • vs. VISTA3D: +2.9% mean DSC.
  • vs. BiomedParse (FLARE22, 3D evaluation): CL-NetU5 90.7% DSC vs. BiomedParse 23.7% DSC (+67.0%).
  • Notable per-organ gains over nnUNet ensemble: hypothalamus +13.9% DSC, brachial plexus +13.8%, chiasm +11.9%, optical nerves +11.8%.
  • External evaluation (7 unseen datasets, 1,979 scans, 43 anatomies): CL-NetU36 consistently outperforms nnUNetE36, MultiTalent, and all SAM-style models.
  • Pruning validation: 99/101 decoders pruned >90%; DSC delta post-pruning within ±0.1% for most decoders.

Limitations

  • Private datasets (65% of training data) are not publicly released, limiting full reproducibility.
  • Continual segmentation mode shows modestly larger performance gaps for head & neck (1.0% DSC gap vs. universal mode) and lesions (0.20mm ASD gap), indicating these remain harder under sequential training.
  • Lesion segmentation ASD remains high (3.72mm) due to ambiguous boundaries and irregular shapes; this category lags other anatomy groups.
  • Model is CT-only; no extension to MRI, PET, or other modalities.
  • 73 novel fine-grained anatomies lack external validation datasets, so their generalization is assessed only internally.
  • LTH pruning requires full decoder training before compression; the iterative pruning pipeline adds engineering complexity.

Relevance to Foundation Models in Medicine

CL-Net directly challenges the dominant SAM-style "segment anything" paradigm for medical imaging, demonstrating that purpose-built continual learning architectures outperform prompt-based foundation models on 3D CT segmentation by margins exceeding 9.9–67% Dice — a direct empirical refutation of the assumption that scale and generality of SAM-derived models transfer well to volumetric medical data. Its frozen-encoder / pruned-decoder design provides a scalable alternative blueprint for building comprehensive medical segmentation foundations without the optimization difficulties that plague joint multi-dataset training. The 235-anatomy coverage and 55.5 MB footprint make CL-Net a practical backbone for downstream clinical AI tasks (radiation oncology planning, opportunistic screening, disease quantification), positioning it as infrastructure for the medical foundation model ecosystem rather than a monolithic foundation model itself.