Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy

🕒 Published (v1): 2026-07-10 06:47 UTC · Source: Arxiv · link

Why this paper was selected

Generalist-specialist synergy for medical image understanding; addresses known VLM accuracy gap in clinical settings

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SuG (Super-Generalist) is a medical image understanding framework that fuses specialist segmentation supervision with generalist vision-language learning to simultaneously achieve broad disease coverage, specialist-competitive diagnostic accuracy, and interpretable lesion grounding on 3D CT. It outperforms both pure generalists and specialist-generalist synergy baselines on chest and abdominal CT benchmarks, and even surpasses specialist-only models (nnUNet) on tumor diagnosis tasks.

Problem

Vision-language generalist models for medical imaging cover many diseases but lack fine-grained anatomical and lesion awareness, yielding poor lesion localization and diagnostic accuracy inferior to task-specific specialists. Conversely, supervised specialist models achieve strong lesion-level performance but cannot generalize across anatomies or unseen disease classes. No existing framework resolves this trilemma of broad scope, high accuracy, and interpretable grounding simultaneously.

Method

SuG augments a 3D vision-language backbone (vision encoder \(\psi\), text encoder \(g_{\theta_\text{text}}\)) with a specialist decoder \(\phi\) trained on voxel-level annotations through three segmentation heads:

1. Anatomy segmentation predicts \(Z_j^\text{anat} \in \mathbb{R}^{C_\text{anat} \times D \times H \times W}\).
2. Class-specific (CS) lesion segmentation predicts per-lesion-class logits, masked by a binary class indicator \(C_j\) to handle single-class annotation per sample: \(L_\text{CS} = L_\text{seg}(C_j \odot Z_j^\text{CS},\, C_j \odot M_j^\text{CS})\).
3. Class-agnostic (CA) lesion segmentation merges all lesions into one foreground class and masks the loss with a validity mask \(V_j\) to avoid treating unlabeled anatomies as negatives: \(L_\text{CA} = L_\text{seg}(Z_j^\text{CA} \odot V_j,\, M_j^\text{CA} \odot V_j)\).

Specialist-enhanced vision-language alignment uses predicted anatomy masks to extract and normalize multi-scale anatomy-specific visual tokens \(a_{j,t}\), aligns them with per-anatomy text embeddings \(t_{j,t}\) (anatomy sentences extracted from radiology reports via Qwen LLM) via anatomy-wise contrastive loss \(L_\text{generalist}\) (Eq. 9).

Lesion-guided attention calibration computes text-conditioned sigmoid attention \(\alpha_{j,q}^{(s)} = \sigma(h_j \cdot f_{j,q}^{(s)})\) at each spatial position and supervises it with binary cross-entropy against the resized lesion mask, enforcing pathology-focused visual attention (Eq. 10).

Training follows a three-stage progressive curriculum: Stage 1 — specialist only; Stage 2 — specialist + generalist alignment; Stage 3 — all three losses.

Key Contributions

• First framework to formalize Generalist–Specialist (S+G) synergy for medical imaging, resolving the three-way trade-off between scope, accuracy, and grounding.
• Dual-mode lesion segmentor: class-specific head for supervised lesion types; class-agnostic head (with validity masking) for generalization to unannotated lesion categories.
• Specialist-enhanced vision-language alignment: mask-guided multi-scale anatomy token extraction with anatomy-wise contrastive learning.
• Lesion-guided cross-modal attention calibration: supervision of text-conditioned attention maps with predicted lesion masks as spatial priors.
• State-of-the-art across 10 CT benchmarks; surpasses nnUNet specialist on several tumor tasks.

Results

• MedVL-CT69K (54 diseases, 15 anatomies): SuG 90.1% avg AUC vs. best S+G baseline VLWS 86.7% (+3.4%) and best generalist ViSD-Boost 84.9%.
• Merlin benchmark: 85.2% AUC / 80.8% F1 (balanced) and 85.1% AUC / 60.7% F1 (full), surpassing all baselines.
• CT-RATE (chest): 81.1% AUC; RAD-ChestCT (external): 73.4% AUC, beating HCFNet by +0.8% and +1.7% respectively.
• Fine-grained tumor diagnosis (LesionSegAbdomen): SuG 96.1% avg AUC vs. nnUNet-fuse 94.7% — surpasses the specialist.
• Lesion grounding (LesionSegAbdomen): SuG 90.8% AUC / 59.5% AUPR / Cohen's \(d\) = 2.9 vs. generalist baseline 47.0% AUC / 13.1% AUPR / \(d\) = −0.1.

Limitations

• Evaluated exclusively on CT modality; generalization to MRI, X-ray, or pathology images is not demonstrated.
• Specialist decoder requires voxel-level annotation datasets (\(D_S\)), which are expensive and anatomy-specific; coverage of unannotated anatomies still depends on the class-agnostic head's validity mask heuristic.
• Anatomy text extraction relies on an external LLM (Qwen) applied to radiology reports; errors in this extraction could degrade alignment quality.
• External validation is limited to one chest CT dataset (RAD-ChestCT); abdominal external validation is absent.
• The three-stage progressive training increases computational cost and complexity compared to end-to-end baselines.

Relevance to Vision-Language Models

SuG directly advances medical VLMs by demonstrating that injecting voxel-level spatial priors from segmentation experts into cross-modal contrastive learning substantially improves both diagnostic accuracy and spatial interpretability — two persistent failure modes of general-purpose VLMs in clinical settings. The anatomy-wise contrastive alignment and attention calibration strategies are transferable design patterns for any VLM operating on dense prediction tasks where region-specific grounding matters. The class-agnostic lesion head additionally shows how VLMs can be extended to zero-shot spatial generalization beyond training categories, relevant to open-vocabulary grounding research. The work thus bridges the gap between CLIP-style global alignment and the fine-grained localization demanded by real-world vision-language applications in high-stakes domains.