Skip to content

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports

🕒 Published (v1): 2026-07-02 09:08 UTC · Source: Arxiv · link

Why this paper was selected

Multi-image ultrasound VLM with long-form reports; practical medical LVLM with realistic data regime

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LUMI is a large vision-language model (LVLM) for real-world ultrasound report generation, trained on 1.5M multi-image examination cases paired with uncurated clinical reports. By aligning data at the examination level—multiple images per exam matched to one long-form report—and fine-tuning Qwen3-VL-4B via LoRA, LUMI achieves state-of-the-art performance across five examination types without any task-specific architectural modifications.

Problem

Existing ultrasound LVLMs are trained on single-image or single-organ settings, which diverges sharply from clinical workflows where an exam spans dozens of images across multiple organs and generates a detailed long-form report. This mismatch limits both generalization and clinical fidelity. More fundamentally, it is unclear whether progress in ultrasound LVLMs requires novel architectures or whether data scale and clinically faithful supervision suffice.

Method

The authors construct a dataset of 1,464,461 real-world ultrasound examinations (17,673,019 images; avg. 12.1 images/exam) spanning 14+ anatomical structures across five clinical domains (thyroid, breast, upper abdomen, gynecology, male urinary), collected from multiple institutions between 2014–2025 and de-identified. Data is organized at the examination level: all images from a single exam are grouped and paired with the corresponding clinical report.

A standard Qwen3-VL-4B-Instruct backbone is fine-tuned via LoRA (\(r=16\), \(\alpha=32\), dropout 0.1) inserted into the self-attention projections (q/k/v/o_proj) of the decoder. The model is trained with autoregressive cross-entropy loss for 182,936 iterations using AdamW (\(\text{lr}=10^{-4}\), cosine schedule, weight decay 0.01, gradient clipping at 1.0), with a 4,096-token context and bfloat16 mixed precision.

Clinical fidelity is assessed via an LLM-based F1 evaluator that extracts lesion-level findings from both generated and reference reports and computes precision, recall, and F1 per lesion—complementing standard NLG metrics (BLEU-1/2/3/4, ROUGE-L, METEOR, BERTScore).

Key Contributions

  • A large-scale, examination-level ultrasound dataset: 1.46M exams, 17.7M images, 14+ organs, paired with uncurated clinical reports from real-world practice.
  • Empirical demonstration that data scale and examination-level alignment—not architectural complexity—are the decisive factors for ultrasound LVLM performance.
  • LUMI, a LoRA-finetuned Qwen3-VL-4B achieving state-of-the-art on NLG and clinical F1 metrics across five examination categories, outperforming larger and more complex medical-domain models.
  • Model and data scaling analyses: F1 gains saturate between 4B and 8B parameters, but clinical F1 shows no saturation at 1.5M training samples, pointing to data as the primary bottleneck.

Results

  • Overall: LUMI ranks first on all metrics in the AVG. column across five examination types (BLEU-1/2/3/4, ROUGE-L, METEOR, BERTScore, clinical F1).
  • AVG. BL-1: LUMI 0.789 vs. best baseline InternVL3.5-30B-A3B 0.343; EchoVLM (ultrasound-specific) 0.323.
  • AVG. F1: LUMI 0.705 vs. EchoVLM 0.233; Qwen3.5-27B 0.307; Lingshu-32B 0.194.
  • AVG. BERTScore: LUMI 0.920 vs. EchoVLM 0.731; InternVL3.5-38B 0.731.
  • Model scaling: F1 improves from 0.614 (2B) to 0.705 (4B), but 8B yields no substantial further gain.
  • Data scaling: Clinical F1 continues rising through the full 1.5M training split with no sign of saturation; NLG metrics (especially BERTScore) plateau earlier.

Limitations

  • LUMI may hallucinate findings at inference time when presented with severely incomplete image sets, since training always pairs the full exam image set with the report.
  • Scaling beyond 4B parameters yields negligible benefit under the current training setup, suggesting training recipe or data diversity may be the bottleneck rather than model capacity.
  • Reports are in Chinese; cross-lingual and cross-institutional generalization is not assessed.
  • Evaluation relies on LLM-based F1 for clinical correctness, which introduces its own model-dependent noise rather than clinician adjudication.
  • Dataset is proprietary and not publicly released, limiting reproducibility.

Relevance to Vision-Language Models

This paper provides strong empirical evidence that examination-level, multi-image supervision with large-scale domain-specific data can unlock VLM capability in a modality (ultrasound) previously resistant to standard LVLM approaches. For researchers tracking VLMs in medical imaging, LUMI demonstrates that the conventional single-image fine-tuning paradigm systematically underutilizes clinical context and that LLM-based clinical F1 metrics expose gaps invisible to NLG metrics alone. The scaling analysis—data still not saturated at 1.5M, model saturated at 4B—offers a concrete design principle for future ultrasound and multi-image medical VLMs. It also sets a rigorous new baseline against which future architecture or training innovations (e.g., CoT, segmentation heads) must compete.