Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models¶

🕒 Published (v1): 2025-12-15 08:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PRIMED is a continual learning (CL) framework for generalist medical VLMs that integrates retrieval-augmented generation (RAG) into knowledge distillation to combat catastrophic forgetting. It builds an 18-million-entry multimodal PubMed retrieval database and a 3,000-question pool to dynamically construct reference datasets, enabling fine-grained intra-domain memory while bridging large cross-domain gaps. The method achieves state-of-the-art performance across a new Medical Generalist Task Incremental Learning (MGTIL) benchmark.

Problem¶

Existing continual learning strategies for VLMs rely on replay from ImageNet-sampled or generative synthetic data, which is ill-suited for medical VLMs built around caption-based supervision and clustered, domain-heterogeneous image distributions. Medical images are intra-domain uniform but inter-domain radically distinct (e.g., pathology vs. CT vs. fundus), requiring simultaneous fine-grained intra-domain discrimination and large cross-domain gap bridging—a dual challenge unaddressed by prior single-domain medical CL or general-purpose CL benchmarks.

Method¶

PRIMED operates on BiomedCLIP (ViT-B/16) as backbone and introduces three coupled components:

Dynamic Multi-stage Retrieval (Dynamic Siphon): Before each task, queries drawn from a hierarchically organized 3,000-question pool and prior task labels retrieve image-caption pairs from an offline 18M-entry database (built from PubMed via Qwen3-Embedding-8B). Retrieval is three-stage: dense embedding cosine similarity → VLM-based cross-modal reranking with deduplication → BM25 lexical gate. The question pool is partitioned into task-specific (𝑀_task), domain-related (𝑀_domain), and general (𝑀_gen) subsets, each retrieved at distinct top-k ratios, producing a personalized per-task reference dataset that shifts from uniform to real-time multimodal as tasks accumulate.
Contrastive Knowledge Transfer (CKT) + Cross-Modality Consistency (CMC): CKT distills from the immediately preceding task's model (teacher) using KL divergence between image-to-text and text-to-image similarity distributions over retrieved data. CMC applies contrastive learning against the identity matrix on the same retrieved batch to maintain cross-modal alignment, compensating for the teacher's own forgetting.
Dynamic Fisher Weight Guard (DFG): Fisher Information Matrix (FIM) diagonals are computed from the combined distillation+alignment loss gradient (excluding CE), yielding dynamically updated per-parameter importance weights that modulate an L2 regularization penalty, outperforming static EWC.

Total training loss: \(\mathcal{L} = \mathcal{L}_{CE} + \alpha\mathcal{L}_{CKT} + \beta\mathcal{L}_{CMC} + \mathcal{L}_{DFG}\).

Key Contributions¶

18M-entry multimodal medical retrieval database from PubMed processed with Qwen3-Embedding-8B, paired with a 3,000-entry hierarchical medical question pool enabling fine-grained domain/task/lesion-level queries.
PRIMED framework: first integration of multi-modal, multi-layer RAG into continual learning for medical VLMs, enabling dynamic reference dataset construction without access to original training data.
Dynamic Fisher Weight Guard: FIM computed from distillation+alignment gradients rather than task loss, providing adaptive regularization that outperforms standard L2 and EWC.
MGTIL benchmark: two evaluation tracks—HieraMedTransfer (9 datasets across X-ray/pathology/fundus for intra- and inter-domain CL) and MedXtreme (6 high-difficulty domains with up to 33 classes per task, using ACC/AUC/BWT metrics).

Results¶

HieraMedTransfer (vs. l2 baseline): - PRIMED_dyn Order I: Transfer +4.7%, Avg. +4.6%, Last +8.0% over l2 baseline; best among all comparators including ZSCL, MoE-CL, DIKI, GIFT. - PRIMED_dyn Order II: Transfer +2.7%, Avg. +4.1%, Last +14.4% over l2 baseline—largest Last improvement by a substantial margin.

MedXtreme (vs. l2 baseline): - PRIMED_dyn Order I: ACC +7.5%, AUC +4.5%, BWT improvement +7.3 points (BWT −2.7 vs. −10.0 for baseline). - PRIMED_dyn Order II: ACC +10.8%, AUC +5.1%, BWT improvement +11.1 points (BWT −3.4 vs. −14.5 for baseline). - PRIMED_dyn outperforms all prior methods (LwF, iCaRL, WiSE-FT, ZSCL, MoE-CL, SND, DIKI, GIFT) on every metric across both benchmarks and both task orders.

Ablation: - Hierarchical retrieval outperforms BM25-only (+3.2% Transfer, +0.9% Avg., +0.3% Last) and embedding-only methods. - DFG outperforms EWC (+2.0% Transfer, +1.8% Avg., +1.5% Last) and l2 (+1.1% Transfer, +0.2% Avg., +0.3% Last). - Full model (CKT+CMC+DFG) outperforms any two-component subset across all metrics.

Limitations¶

Relies on an offline 18M PubMed retrieval database; construction requires significant compute and the Qwen3-Embedding-8B model; database must be rebuilt if the medical literature scope changes.
Teacher model is the immediately preceding task's checkpoint, which itself suffers forgetting—CMC partially compensates but does not fully resolve the cascading teacher degradation.
Evaluated exclusively on BiomedCLIP (ViT-B/16) backbone in the main experiments; generalization to generative or decoder-based medical VLMs is undemonstrated in the main paper.
MGTIL covers classification only; segmentation and report generation tasks are out of scope.
The dynamic retrieval ratio hyperparameters (a, b, c in Dynamic Siphon) require tuning per domain split.

Relevance to Foundation Models in Medicine¶

This work directly addresses one of the central deployment challenges for generalist medical foundation models: how to update a pre-trained VLM on new clinical tasks without retraining from scratch or maintaining per-task models, all while preserving zero-shot capability. The RAG-augmented distillation paradigm is a practical mechanism for privacy-compliant, data-efficient lifelong learning—critical in medical settings where original training data cannot be replayed. The MGTIL benchmark fills a notable gap by providing a rigorous cross-domain, multi-difficulty evaluation suite for medical CL, which will serve as a reference point for future work on continually-updated medical foundation models. The approach is directly extensible to larger medical VLMs (e.g., BiomedCLIP successors, Med-Flamingo) and complements ongoing work on parameter-efficient fine-tuning and modality-specific adaptation.