An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering¶

🕒 Published (v1): 2026-07-13 18:08 UTC · Source: Arxiv · link

Why this paper was selected

Empirical continual-learning study for medical VQA; practical for multi-task clinical VLM deployment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper benchmarks five continual learning (CL) strategies—Replay, EWC, KD, SPG, and O-LoRA—applied to a frozen MedGemma-4B backbone with LoRA adapters on five structurally heterogeneous MedVQA tasks. All tested CL methods partially reduce catastrophic forgetting but none achieves a robust stability-plasticity balance, with performance highly sensitive to task ordering. LoRA angular drift analysis reveals that tighter weight alignment correlates with lower forgetting.

Problem¶

Prior CL work in medical imaging addresses domain-incremental settings where tasks share the same objective but differ in modality or data source. Real clinical workflows, however, require VLMs to sequentially learn tasks with fundamentally different objectives (classification vs. object detection vs. free-form report generation), supervision granularities, and output formats. This structural heterogeneity creates cross-task interference patterns not captured by existing benchmarks, leaving a critical gap in understanding CL behavior for heterogeneous MedVQA.

Method¶

The study uses MedGemma-4B-IT as a frozen backbone; only LoRA parameters (\(r=16\), \(\alpha=32\)) are updated throughout sequential training. All CL methods are adapted to operate solely within the LoRA adapter space. The evaluation platform is the FLARE-MLLM-2D benchmark (19 public medical datasets, 8 imaging modalities, 5 task types):

Classification — Balanced Accuracy
Multi-label classification — Micro-averaged F1
Detection — F1 at IoU 0.5
Cell counting — Mean Absolute Error (MAE)
Report generation — GREEN score

Three task orderings are tested: Order A (Easy→Hard: classification → multi-label → detection → counting → report generation), Order B (Hard→Easy), and Order C (Mixed). Two non-CL references are included: MTL (upper bound, joint training) and SeqFT (lower bound, no forgetting mitigation). Forgetting is quantified as the Forgetting Ratio:

\[\text{FR}^{(t)} = \frac{\max_{t \le i \le T}(p_i) - p_T}{\max_{t \le i \le T}(p_i)} \quad (\text{higher-is-better metrics})\]

LoRA weight drift is analyzed via the Frobenius inner-product cosine similarity \(\text{sim}(t_1, t_2) = \frac{\langle t_1, t_2 \rangle_F}{\|t_1\|_F \|t_2\|_F}\) between consecutive task adapter directions, visualized as polar-mapped 2D arrows.

Key Contributions¶

First systematic empirical study of CL for structurally heterogeneous MedVQA (spanning 5 task types with differing output formats and supervision granularities).
Adaptation of five CL strategies (Replay, EWC, KD, SPG, O-LoRA) to operate exclusively on LoRA parameters in a frozen-backbone setting.
Task-ordering sensitivity analysis across three curricula, identifying report generation as a dominant interference source.
Quantitative visualization of LoRA angular drift and its correlation with observed forgetting patterns per method.

Results¶

MTL (MedGemma-4B) upper bound: Balanced Acc 0.61, multi-label F1 0.55, detection F1 0.16, MAE 261.26, GREEN 0.72.
SeqFT (Order A): catastrophic forgetting across all tasks; classification Balanced Acc 0.16, FR=0.94; detection FR=1.00.
EWC achieves the most consistent stability-plasticity balance: classification Balanced Acc up to 0.52, FR as low as 0.20 (Order C); EWC and KD are most robust to task ordering permutation.
O-LoRA exhibits the highest forgetting: classification FR=1.00 (Orders A and C), multi-label FR=1.00 (Orders A and C), GREEN FR=0.99–1.00 in two orders.
KD achieves FR=0.00 on detection and multi-label for Order A, but shows reduced plasticity on counting and report generation.
Detection F1 remains ≤0.16 across all methods and orders, consistent with prior findings on this benchmark regardless of backbone or fine-tuning strategy.
Report generation introduced early (Orders B, C) amplifies forgetting across all methods by shifting the model toward long-form autoregressive outputs; KD is least affected due to output-level distillation constraining decoding shifts.
LoRA angular drift: Replay shows the broadest angular spread; EWC and KD maintain tighter distributions; O-LoRA shows progressive cumulative directional drift.

Limitations¶

Experiments are limited to a single backbone (MedGemma-4B-IT); behavior on larger VLMs is not evaluated.
LoRA rank is fixed at \(r=16\); the effect of rank on the stability-plasticity trade-off in CL is unexplored.
Detection performance is universally poor (F1 ≤ 0.16), likely a task formulation issue with the FLARE-MLLM-2D benchmark rather than a CL-specific finding.
No task identity is provided at inference, making the setting challenging but conflating task-agnostic generalization with forgetting measurement.
Replay buffer is limited to 100 samples per task; larger buffers are not evaluated.
Only 2D medical imaging tasks are studied; 3D modalities are left for future work.

Relevance to Vision-Language Models¶

This paper directly probes how parameter-efficient fine-tuning (LoRA) of medical VLMs behaves under realistic sequential deployment, where tasks differ not just in data distribution but in output structure—a setting underrepresented in the VLM continual learning literature. The finding that parameter-level constraints alone (EWC, SPG, O-LoRA) are insufficient without functional alignment to preserve output behavior has direct implications for designing deployment-safe VLM update pipelines. The LoRA angular drift analysis provides a mechanistic lens for understanding cross-task interference in adapter-based VLMs, connecting weight-space geometry to empirical forgetting. For VLM researchers, the identification of report generation as a dominant catastrophic forgetting trigger—due to its autoregressive output shift—is a concrete actionable insight for task sequencing and regularization design.