Skip to content

EWC-Guided Diffusion Replay for Exemplar-Free Continual Learning in Medical Imaging

🕒 Published (v1): 2025-09-28 14:23 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Medical imaging foundation models deployed in clinical settings must adapt to new tasks over time, but storing patient data for replay violates privacy regulations. This paper combines class-conditional DDPM generative replay with Elastic Weight Consolidation (EWC) to achieve exemplar-free continual learning, reducing forgetting by >30% relative to DER++ on CheXpert while approaching joint-training performance (0.851 vs. 0.869 AUROC).

Problem

Continual adaptation of medical imaging models is blocked by two competing constraints: (1) privacy regulations prohibit storing patient exemplars for replay, and (2) sequential fine-tuning causes catastrophic forgetting. Existing alternatives fall short—regularization alone degrades under distribution shift, GAN/VAE replay cannot faithfully reproduce fine medical detail, and dynamic expansion is computationally prohibitive.

Method

The framework pairs three components under a unified objective: DDPM replay, EWC regularization, and a lightweight ViT classifier. At each task, a class-conditional DDPM (U-Net backbone, 1000 timesteps, cosine/linear β schedule) is trained to model p(x|y) without storing raw data; 256 class-balanced synthetic samples are placed in a fixed 100 MB replay buffer. The ViT classifier (6 layers, 8 heads, hidden dim 512, patch size 16) is then trained on a 1:1 mix of real and replayed samples. EWC penalizes Fisher-weighted parameter drift from anchored past weights: L_total = CE(fθ(x), y) + λ Σ_i F_i(θ_i − θ_i)². The design is grounded theoretically: average forgetting is bounded as F̄ ≤ α·KL(p_k‖p̂_k) + β·‖θ_K − θ_k‖², linking forgetting to replay divergence (reduced by DDPM) and parameter drift (penalized by EWC). A Fisher Scheduled Replay variant directs generative samples toward tasks with highest KL divergence.

Key Contributions

  • Exemplar-free continual learning framework combining class-conditional DDPM replay and EWC for privacy-preserving adaptation of medical imaging models.
  • Unified 2D/3D handling with a single diffusion model architecture (3D tasks use volumetric convolutions with identical conditioning scheme).
  • Theoretical forgetting bound decomposing forgetting into replay divergence (KL term) and Fisher-weighted parameter drift, empirically validated via joint regression.
  • Demonstration that both bound terms correlate independently with observed forgetting, providing actionable diagnostics for when and why degradation occurs.

Results

  • CheXpert: DDPM+EWC achieves 0.851 AUROC, forgetting F=10.9; DER++ achieves 0.838 AUROC with F=13.8; joint training upper bound is 0.869 AUROC. Forgetting reduction >30% relative to DER++.
  • MedMNIST-2D: 78.1% avg accuracy, F=10.5, AUC=0.866 vs. DER++ at 75.6%, F=14.2, AUC=0.853.
  • MedMNIST-3D: 74.0% avg accuracy, F=12.9, AUC=0.849 vs. DER++ at 70.9%, F=16.8, AUC=0.835.
  • Ablation: removing DDPM increases forgetting by 5.8–7.4%; removing EWC reduces early-task accuracy by 3.7–4.9%, confirming complementary roles.
  • Task order robustness: DDPM+EWC drops only 0.7% accuracy under task sequence reversal vs. 3.1–3.3% for DER++ and SPM.
  • Low-shot (10% data): DDPM+EWC 54.8% vs. DER++ 38.5%, demonstrating replay as implicit augmentation.
  • Forward/backward transfer: FWT=+0.062, BWT=−0.017 vs. DER++ FWT=−0.017, BWT=−0.093.

Limitations

  • Evaluated only on low-resolution MedMNIST proxies and CheXpert; no validation on full-resolution multi-site datasets, MRI, or histopathology.
  • Assumes fixed canonical task order and balanced replay buffers; real clinical streams have class imbalance, evolving label taxonomies, and curriculum drift.
  • DDPM sampling is computationally expensive (3–6 hours per task on A100); generator distillation not yet investigated.
  • Hybrid exemplar baselines (small stored buffer + generative replay) not benchmarked.
  • No explicit uncertainty calibration; fairness under imbalanced replay classes unaddressed.
  • Unified single-DDPM variant reduces storage ~45% but degrades fidelity notably for 3D and chest X-ray tasks (Adrenal3D FID rises from 10.2 to 16.5).

Relevance to Foundation Models in Medicine

This work directly addresses the deployment gap for medical imaging foundation models: how to continually update a pretrained model as clinical distributions shift without violating patient privacy or incurring full retraining costs. The theoretical forgetting bound is particularly valuable as it provides a principled diagnostic framework that can be applied to audit any continual FM update pipeline, relevant to governance and compliance requirements (GDPR, HIPAA). Compared to prior generative replay work using GANs/VAEs, the DDPM approach better preserves fine-grained medical features (sharper cytoplasm boundaries in BloodMNIST, anatomical contours in 3D volumes, rib edges in CXR), which is a prerequisite for replay to be effective in clinical domains. The gap to joint training remains small (~1.8% AUROC on CheXpert), suggesting this is a practical route for privacy-aware continual adaptation of deployed clinical models.