Skip to content

UNICON: UNIfied CONtinual Learning for Medical Foundational Models

šŸ•’ Published (v1): 2025-08-19 17:31 UTC Ā· Source: Arxiv Ā· link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

šŸ’¬ Ask ChatGPTāœ¦ Ask Claude

TL;DR

UNICON is a unified continual learning framework that sequentially adapts a single medical imaging foundation model across new tasks, modalities, and anatomical regions without catastrophic forgetting. Starting from a chest CT classification model (CT-CLIP), it incrementally adds prognosis prediction, CT segmentation, and CT+PET segmentation using lightweight parameter-efficient adapters. The framework eliminates the need for separate specialized models while maintaining performance across all previously learned capabilities.

Problem

Existing medical foundation models are locked to their initial training domain (modality, anatomy, task), and adaptation approaches address only one dimension at a timeā€”new classes, new domains, or new modalitiesā€”in isolation. No prior unified framework enables sequential expansion across all three dimensions simultaneously while avoiding catastrophic forgetting and handling variable image resolutions common in real-world clinical data.

Method

UNICON freezes the base foundation model's encoders (image encoder E_img, text encoder E_text) and incrementally attaches lightweight task-specific adaptation modules A^(m) at each continual learning step. Two complementary adaptation mechanisms are used:

  • Within-Model Adaptation (WMA): LoRA low-rank matrix decomposition (W' = W + Ī¦ā‚Ī¦ā‚‚, rank r ā‰Ŗ min(d,h)) injected into the pretrained encoder weights.
  • Post-Model Adaptation (PMA): Nonlinear MLP layers (including a multimodal fusion MLP) appended after the encoder, plus a decoder for segmentation reconstruction.

Resolution adaptation is handled by adding and fine-tuning task-specific patch embedding layers and positional embeddings for each new domain, allowing the model to generalize beyond its original fixed resolution.

At inference, the correct adapter set is selected dynamically based on the input modality and target task. Experiments use CT-CLIP (chest CT classifier) as the base, sequentially adapted on the HECTOR dataset (488 head-and-neck tumor patients, 7 centers): Step 1 = prognosis (CT+PET+EHR), Step 2 = CT segmentation, Step 3 = CT+PET segmentation. EHR data is converted to report-like text via GPT-4 prompt engineering before being fed to the text encoder.

Key Contributions

  • Unified continual learning framework enabling simultaneous cross-task, cross-modality, and cross-anatomy adaptation of a medical FM without modifying base parameters.
  • Resolution adaptation mechanism via learnable task-specific patch embeddings, enabling deployment on variable-resolution scans.
  • Demonstration that a chest CT classification FM (CT-CLIP) can be continually extended to head-and-neck prognosis, CT segmentation, and CT+PET segmentation without catastrophic forgetting.

Results

  • Prognosis (C-Index): UNICON (image+text, PMA+WMA) achieves 0.721 with DeepHit vs. baseline 0.661 (ā‰ˆ+6% absolute); text-only PMA alone reaches 0.686. Image-only adaptation yields no meaningful improvement (0.566ā€“0.626 vs. 0.661 baseline), attributed to EHR-heavy task nature.
  • CT Segmentation (Dice): UNICON CT adaptation (PMA+WMA) reaches 0.628 vs. UNETR baseline 0.609 (+2% Dice).
  • CT+PET Segmentation (Dice): UNICON with PMA+WMA reaches 0.657 vs. CT-only baseline 0.609 (+5% Dice over CT-only baseline, as stated in abstract).
  • The adapted model spans 4 tasks (classification, prognosis, CT-seg, CT+PET-seg) from a single base FM; the original classification capability is retained.

Limitations

  • Evaluated on a single dataset (HECTOR, 488 patients, head-and-neck only); generalizability to other anatomies or modalities is unvalidated.
  • The amount of data required to adapt to a new domain or task is not quantifiedā€”a key practical consideration for data-scarce clinical settings.
  • No explicit catastrophic forgetting measurement reported (e.g., backward transfer metrics); retention is inferred from the base model's frozen encoders rather than empirically measured on held-out original-task data.
  • Baseline comparisons are limited; no comparison against other continual learning strategies (EWC, replay-based methods) that also target forgetting prevention.
  • Relies on GPT-4 for EHR-to-text conversion, introducing an external dependency and potential information loss.

Relevance to Foundation Models in Medicine

UNICON directly addresses a core scalability bottleneck for medical FMs: the one-model-per-task fragmentation that results from narrow pre-training scopes and data scarcity. By demonstrating that parameter-efficient adapters (LoRA + MLP heads) can extend a radiology FM across modalities, anatomies, and tasks sequentially, it advances the case for continually evolving generalist medical AI rather than siloed specialist models. This connects tightly to the broader effort to build universal medical imaging FMs (e.g., BiomedParse, CT-CLIP, universal segmentation models) by providing a principled adaptation protocol that preserves foundational representations. The resolution adaptation mechanism also addresses a practical deployment barrier often overlooked in FM papers.