Skip to content

Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

🕒 Published (v1): 2026-06-18 16:55 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Why this paper was selected

MICCAI 2026; 1.2M CT/MR grounded pairs; scalable spatially grounded radiology VLM recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RadGrounder is a PaliGemma 2-based radiology VLM trained on RefRad2D, a 1.2M-pair bilingual (German/English) CT/MRI dataset with automatically derived spatial grounding annotations, requiring no manual labeling. It jointly performs report generation, VQA, and bounding-box/segmentation grounding. Token-based bounding-box detection achieves effective localization without degrading language task performance.

Problem

Radiology VLMs generate fluent text but cannot reliably ground outputs in specific image regions, limiting clinical verifiability and enabling hallucinations. Training grounded VLMs on CT/MRI is further constrained by data scarcity — prior 2D slice efforts topped out at ~4,009 images — and prior spatially grounded medical VLMs have focused almost exclusively on chest X-rays rather than CT/MRI at scale.

Method

RefRad2D dataset construction: 1.2M image–text pairs from 945k CT + 321k MRI slices (one decade of clinical routine at a university hospital). Captions are rewritten by GPT-OSS (120B) to strip longitudinal references, and German reports are translated to English by Gemma 3 (27B) with iterative LLM-judge prompt refinement. For grounding, TotalSegmentator generates 3D segmentation masks for 117 CT / 50 MRI classes (harmonized to \(C=121\)). GPT-OSS extracts anatomical mentions from each caption and maps them to the same schema; a triplet \((I, T, M^T_I)\) is valid only if \(C_T \cap C_I \neq \emptyset\), yielding 236,157 grounded slice–text pairs. For VQA, Gemma 3 (27B) synthesizes ~9.6M QA pairs (open, yes/no, multiple choice).

RadGrounder architecture: Built on PaliGemma 2 (3B) — SigLIP-So400m vision encoder + Gemma-2B decoder. Two grounding strategies are ablated: (1) token-based bounding-box detection, where coordinates are discretized into 512 bins and generated autoregressively as text tokens in the format <p bbox> [LOC] id=<segID> KEYWORD </p>; (2) an auxiliary segmentation head that triggers on a </seg> token, projects the hidden state to a 256-dim prompt, and drives a Two-Way Transformer to produce a binary mask with loss \(\mathcal{L} = \mathcal{L}_\text{txt} + \lambda_\text{seg}(\mathcal{L}_\text{focal} + \mathcal{L}_\text{dice})\). Multi-task training alternates batches across report generation, VQA, and grounding.

New metrics: LLMScore (Gemma 3 as judge, 5-point scale normalized to [0,1]; \(r=0.977\) Pearson correlation with radiologist scores, Krippendorff's \(\alpha=0.958\)) and Grounding-IoU (G-IoU), which matches predicted and ground-truth entities by text-embedding cosine similarity before computing spatially weighted IoU, penalizing both hallucinations and omissions.

Key Contributions

  • RefRad2D: 1.2M bilingual CT/MRI image–text pairs with automated LLM-curation pipeline; RefRad2D-Grounded subset of 236,157 spatially annotated triplets; RefRad2D-VQA of ~9.6M pairs — all without manual annotation.
  • RadGrounder: Multi-task VLM jointly performing report generation, VQA, and spatial grounding on 2D CT/MRI slices.
  • Finding: Token-based bounding-box detection achieves higher G-IoU (43.6 vs. 36.9) than the auxiliary segmentation head while adding zero model components or extra loss terms.
  • Finding: Adding grounding supervision does not degrade VQA or report generation quality.
  • LLMScore and G-IoU: Two new evaluation metrics for semantic correctness and spatially-aware grounding in radiology.

Results

  • Slake: RadGrounder achieves F1 87.7, Closed Accuracy 90.3 — comparable to BiomedGPT-B (85.2/89.9) and LLaVA-Med (86.8), surpassing Med-Gemini (75.8/84.8) and MedGemma (72.3/87.6).
  • VQA-RAD: Open F1 50.7, the highest among compared methods (vs. RadFM 14B: -, MedGemma: 20.8, Med-Gemini: 26.0 on same split).
  • Internal grounding: Token detection G-IoU 43.6 vs. segmentation head G-IoU 36.9.
  • Dataset transfer: Adding RefRad2D to external fine-tuning improves Slake F1 (87.7 vs. 86.9) and VQA-RAD Open F1 (50.7 vs. 47.9) over external-only training.
  • Vision encoder: Freezing the medical-domain-adapted SigLIP encoder is on par with unfreezing variants while reducing memory and training time (~2.5 days on a single H100).

Limitations

  • Training data from a single hospital; multi-center validation absent.
  • Spatial grounding targets anatomical structures via TotalSegmentator, not pathological findings — lesion-level localization is not supported.
  • Comparing token-based detection and segmentation grounding via a single metric is nontrivial, as they solve different geometric tasks (bounding box overlap vs. pixel-level mask agreement).
  • Bilingual expansion covers only German↔English; broader language coverage is unaddressed.

Relevance to Vision-Language Models

This work is directly relevant to the line of research on grounded and medically specialized VLMs, demonstrating that large-scale 2D slice supervision — historically bypassed in favor of 3D volumetric or chest X-ray–centric approaches — is a viable and scalable training paradigm. The key insight that token-based coordinate generation suffices for effective spatial grounding (no auxiliary decoder needed) is broadly applicable to any domain where annotated grounding data is scarce. The automated LLM-driven curation pipeline and the proposed LLMScore/G-IoU metrics offer reusable infrastructure for medical VLM development. The no-cost grounding result (spatial supervision does not hurt language quality) is an important empirical finding for multi-task VLM design.