Skip to content

MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence

🕒 Published (v1): 2026-03-28 07:26 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; medical VLM with lesion detection, tracking, and visual explainability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MEDIC-AD is a stage-wise medical VLM that adds explicit anomaly-aware tokens (<Ano>), temporal difference tokens (<Diff>), and a ConvNeXt-based heatmap decoder to a pretrained medical VLM backbone (Lingshu-7B). The three progressive stages encode the clinical workflow—detect, compare, explain—achieving state-of-the-art results on zero-shot lesion detection, longitudinal symptom tracking, and anomaly segmentation.

Problem

Existing medical VLMs (including strong baselines like Citrus-V and Lingshu) are optimized for broad medical knowledge and single-image tasks (VQA, report generation) but lack three capabilities critical for clinical deployment: (1) reliable lesion detection, (2) temporal tracking of disease progression across paired scans, and (3) spatially grounded visual explainability linking model decisions to observable image regions.

Method

MEDIC-AD extends a pretrained medical VLM (Lingshu-7B) through three frozen-then-finetuned progressive stages:

Stage 1 — Anomaly Detection. Learnable <Ano> tokens are produced by an Anomaly Processor: two system tokens (Abnormal/Normal) attend via cross-attention to multi-scale features from four intermediate vision-encoder layers (using Visual Soft Prompt Tuning to preserve pretrained weights). A Sigmoid—not Softmax—computes patch-wise anomaly probabilities; the difference between abnormal and normal attention yields an Anomaly Attention Map that modulates visual features element-wise. 2D global pooling then feeds an Anomaly Q-Former + 2-layer MLP to produce <Ano> tokens, which are concatenated with visual and text embeddings.

Stage 2 — Difference Reasoning. Given prior and current image pairs, Stage 1 produces anomaly-augmented features \(V_1^*\) and \(V_2^*\). A Diff Q-Former contrasts these to produce <Diff> tokens encoding only lesion-specific temporal change, explicitly disentangling pathological progression from acquisition noise. Training uses MIMIC-Diff-VQA.

Stage 3 — Visual Explainability. <Ano> tokens are fused with intermediate vision-encoder feature maps and decoded through a compact ConvNeXt-based segmentation head to produce a spatial heatmap \(M = f_h([f_p(V^*); \langle\text{Ano}\rangle])\). No external decoder (e.g., SAM2) is used.

Key Contributions

  • Stage-wise training framework (detect → compare → explain) that incrementally builds clinical reasoning without catastrophic forgetting of general medical knowledge.
  • <Ano> token mechanism: Sigmoid-gated cross-attention over multi-scale encoder layers producing compact, lesion-discriminative latent tokens.
  • <Diff> token mechanism: Diff Q-Former that disentangles inter-scan pathological change from irrelevant appearance variation.
  • Anomaly-token-guided ConvNeXt heatmap decoder that grounds textual predictions in spatial lesion evidence without requiring SAM2 or external segmentation models.
  • Comprehensive evaluation on real longitudinal hospital data demonstrating deployment-ready clinical reliability.

Results

Zero-shot Anomaly Detection (F1, Tab. 1): - MEDIC-AD Avg. F1: 91.6 vs. Lingshu 88.7, Citrus-V 84.2, GPT-4o 62.4, Claude-3.5 69.2 - Brain MRI F1: 92.0; Head CT F1: 95.5; Br35h F1: 90.2; COVID-19 X-ray F1: 88.5

Symptom Tracking — MMXU benchmark (accuracy, Tab. 2): - MEDIC-AD Overall: 0.655 vs. Lingshu 0.620, Citrus-V 0.571, GPT-4o 0.571, Claude-3.5 0.502

Visual Grounding (Tab. 3, AUC / mIoU): - BraTS2021: MEDIC-AD AUC 99.8 / mIoU 87.6 vs. Citrus-V 98.8 / mIoU 32.6 - RESC: MEDIC-AD AUC 100 / mIoU 97.2 vs. Citrus-V 87.6 / mIoU 2.1 - BTCV+LiTs: MEDIC-AD mIoU 83.6 vs. Citrus-V 1.7 - ChestX-Det: MEDIC-AD AUC 99.8 / mIoU 79.8 vs. Citrus-V 98.0 / 12.4

Limitations

  • Evaluated only on modalities present in the training distribution (Brain MRI, Head CT, Chest X-ray); generalization to other modalities (e.g., retinal OCT, pathology whole-slide) is not fully addressed.
  • Temporal reasoning (Stage 2) is validated solely on chest X-ray pairs (MMXU/MIMIC-Diff-VQA); performance on other longitudinal modalities is not reported.
  • The ablation isolating each stage's contribution is implicit (progressive evaluation per task) rather than an explicit held-out ablation study.
  • Real hospital evaluation appears qualitative/anecdotal; no quantitative metrics on the real-world longitudinal clinical cohort are reported in the provided text.
  • Relies on Lingshu-7B as a backbone; generalizability of the token mechanisms to other base VLMs is untested.

Relevance to Vision-Language Models

MEDIC-AD directly advances medical VLM capabilities by introducing task-specific learnable token types (<Ano>, <Diff>) that inject structured inductive biases into a general-purpose VLM without replacing its architecture—an approach transferable to other grounded-reasoning domains. The stage-wise curriculum, where each stage adds a frozen prior stage's outputs as inputs, offers a practical recipe for extending large pretrained VLMs to multi-step, multi-image, and spatially grounded reasoning without catastrophic forgetting. The substantially improved visual grounding (mIoU gains of 50–80+ points over SAM2-based Citrus-V) suggests that anomaly-token-guided lightweight decoders can match or exceed heavier segmentation backbones. For VLM researchers tracking grounded multimodal reasoning, temporal/multi-image understanding, and explainability, this work provides both strong baselines and a modular extension pattern.