Skip to content

UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation

🕒 Published (v1): 2025-11-27 09:33 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UMind-VL is the first ultrasound-specific foundation model to unify pixel-level grounded perception (segmentation, detection, keypoint localization) with high-level clinical reasoning (diagnosis, view classification) in a single VLM framework. It pairs a lightweight Dynamic Convolutional Mask Decoder conditioned on LLM token outputs with a 1.2M-pair ultrasound dataset (UMind-DS) covering 16 anatomical regions. The result outperforms existing generalist medical VLMs and matches or exceeds specialist models across all evaluated ultrasound tasks.

Problem

The ultrasound domain has a persistent functional split: segmentation-centric models (SAM adaptations such as MedSAM, UltraSAM) deliver pixel-level precision but carry no diagnostic semantics, while medical VLMs (LLaVA-Med, RadFM, DolphinV1) provide reasoning but lack fine-grained spatial grounding. No existing model coherently integrates both, preventing clinically viable workflows that require simultaneous anatomical localization and pathology interpretation. General-purpose VLMs also systematically misidentify basic ultrasound anatomy (Figure 2 evaluation), attributed to absent spatial grounding optimization and lack of ultrasound-specific vision-language alignment data.

Method

UMind-VL follows an encoder–decoder VLM backbone (vision encoder → LLM decoder) augmented with two innovations:

Dynamic Convolutional Mask Decoder: For segmentation, the LLM emits N consecutive <|seg_mask|> special tokens per target. Their LLM-layer feature embeddings (Fq ∈ ℝ^{N×D}) are projected into dynamic convolution kernels (K1, K2). These kernels perform two sequential 2× transposed convolutions over spatially reshaped vision features (H0 ∈ ℝ^{H×W×C}), followed by bilinear upsampling to produce full-resolution binary mask logits. The decoder introduces only three linear projections and a LayerNorm—deliberately lightweight to force the VLM backbone to internalize spatial grounding rather than delegating it to a heavy task-specific head.

Unified spatial token vocabulary: Detection uses <|box_start|>(x1,y1),(x2,y2)<|box_end|>; keypoints use <|point_start|>(x,y)<|point_end|>; geometric measurements use <|line_start|>(x1,y1),(x2,y2)<|line_end|>; segmentation emits <|seg_mask|> tokens consumed by the mask decoder. Cross-token object references (<|object_ref_start|>…<|object_ref_end|>) link spatial outputs to semantic descriptions in subsequent tokens.

UMind-DS: 1.2M ultrasound image–text pairs across 16 anatomical regions (cardiac 29%, thyroid 13%, breast 13%, musculoskeletal 12%, others). Comprises 815k clinician-validated real-world samples and 424k synthetic samples targeting hallucination mitigation and rare-pathology coverage. Includes pixel-level segmentation masks, bounding boxes, keypoint annotations, and chain-of-thought diagnostic rationales. Multi-stage annotation: sonographer annotation → double-blind senior radiologist review → senior expert arbitration for disagreements.

Training uses a joint loss: autoregressive token prediction for all tasks, plus a region-overlap + pixel-level classification loss on the Dynamic Conv Mask Decoder for segmentation samples.

Key Contributions

  • UMind-VL architecture: first single framework to co-optimize Ultrasound Grounded Perception and Ultrasound Comprehensive Interpretation end-to-end.
  • Dynamic Convolutional Mask Decoder: lightweight (3 linear layers + LayerNorm), LLM-conditioned dynamic-kernel design that offloads spatial learning to the VLM backbone rather than relying on a heavy external segmentation head (e.g., SAM).
  • Unified spatial token vocabulary encoding boxes, points, lines, and masks within the standard LLM token sequence.
  • UMind-DS: 1.2M ultrasound image–text pairs with pixel-level annotations and CoT rationales across 16 anatomical regions and three imaging modalities (2D, M-mode, Doppler).
  • Demonstrated strong out-of-distribution generalization beyond the training distribution.

Results

  • Surpasses all evaluated generalist multimodal models across segmentation, detection, keypoint localization, and diagnostic reasoning benchmarks.
  • Segmentation: surpasses Mask2Former (specialist).
  • Detection: rivals Deform-DETR (specialist).
  • Qualitative comparison on standard abdominal US (liver + kidney identification, Figure 2): UMind-VL correctly identifies both organs and provides pixel-level segmentation masks; GPT-4o, Claude Sonnet 4.5, Qwen3-Max, Lingshu-32B, and MedGemma-27B all fail to correctly identify both structures (only Gemini 2.5 Pro partially succeeds among public LLMs).
  • Robust generalization on out-of-distribution datasets (specific metrics not provided in the supplied text).

(Note: full quantitative tables are not included in the provided text excerpt; the above reflects the claims stated in the paper.)

Limitations

  • Technical report, not peer-reviewed; full quantitative result tables absent from the provided text.
  • Proprietary in-house datasets (annotation and sourcing details withheld) limit external reproducibility.
  • Synthetic data generation pipeline is described only partially in the supplied text; hallucination mitigation efficacy is asserted but not quantified.
  • Cardiac dominates the dataset (29%), which may bias generalization across less-represented organs (gastrointestinal: 0.23%, prostate: 0.54%).
  • Evaluation against human sonographer performance is not reported; comparisons are model-to-model.
  • Dynamic Conv Mask Decoder produces single-class binary masks per <|seg_mask|> sequence; complex multi-instance or overlapping structure scenarios are not explicitly addressed.

Relevance to Foundation Models in Medicine

UMind-VL directly addresses one of the central open challenges in medical foundation model research: bridging perception and reasoning within a single architecture rather than coupling isolated specialist modules. Its Dynamic Convolutional Mask Decoder design—conditioning pixel-level outputs on LLM representations while keeping the decoder intentionally lightweight—offers a transferable architectural principle applicable beyond ultrasound to other modalities (pathology, endoscopy) where pixel grounding matters clinically. The dataset construction methodology (1.2M pairs with CoT rationales, multi-stage expert annotation, synthetic hallucination-mitigation samples) provides a template for scaling ultrasound-specific training data, a resource gap that has historically limited this modality compared to radiology. For researchers tracking generalist medical foundation models, UMind-VL demonstrates that modality-specific co-design of data and architecture substantially outperforms adapting general VLMs (LLaVA-Med, MedGemma) to ultrasound.