Skip to content

EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography

🕒 Published (v1): 2026-06-26 14:58 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EchoSonar-R is a multi-view vision-language model that jointly performs multi-label disease classification and report generation from echocardiography studies, producing interpretable reasoning traces grounded in anatomical evidence. It combines a frozen spatiotemporal video encoder with a structure-aware cardiac detector and is trained in two stages (SFT followed by GRPO with task-specific rewards), achieving 67.4% macro balanced accuracy on a private multi-view set and a 0.800 GREEN clinical faithfulness score.

Problem

Automated echocardiography models have focused on single-view or single-task objectives (view classification, segmentation, EF estimation), and existing echocardiography foundation models (EchoCLIP, EchoPrime, PanEcho) primarily map visual inputs to classification labels or retrieve reports without generating explicit cross-view grounded reasoning or clinician-style reports. The lack of interpretable diagnostic reasoning limits clinician trust and clinical adoption. No prior work addresses multi-view reasoning-based report generation and disease classification grounded in anatomical evidence from echocardiography video.

Method

  • Multi-view visual encoding: Each video is processed by a frozen mViT-based spatiotemporal encoder (EchoPrime, 393 tokens/video \(T=393\)) and a frozen RT-DETR-L detector localizing seven key cardiac structures. Detector query embeddings are averaged across frames to produce per-structure representations.
  • Cross-modal projection: Two trainable 2-layer MLPs (GELU) project video tokens and structure tokens into the LLM embedding space (\(d_{LM}\)). The input sequence interleaves projected video tokens, structure tokens, and view-identifier text from a pretrained view classifier.
  • Two-stage training:
  • SFT (3 epochs, LR \(2 \times 10^{-5}\)): autoregressive cross-entropy on reasoning-annotated targets across five question types (full report, conclusion-only, structure description, binary classification, multi-label). Reasoning traces \(R^*\) generated by prompting Qwen3-8B with guideline-based templates. Full LM parameters unfrozen.
  • GRPO (5 epochs, LR \(1 \times 10^{-6}\), LoRA \(r=64\), \(G=8\) completions): optimizes a composite reward \(r = r_{\text{fmt}} \cdot r_{\text{cor}} + r_{\text{len}}\) — format reward (enforces </think> tag), correctness reward (IoU for multi-label, per-section cosine similarity via EchoPrime text encoder for reports), and length penalty (\(L_{\min}=400\)). KL divergence penalty is omitted.
  • LLM: Qwen3-8B; visual encoders frozen throughout.
  • Data: Private dataset (5,061 train / 1,215 test studies, median 7 views, 12 abnormality classes, prevalence 2.5%–65.0%). Detector trained on 2,689 studies with bounding box annotations propagated via RAFT optical flow.

Key Contributions

  • First multi-view reasoning-enabled VLM for echocardiography that jointly performs disease classification and report generation with explicit reasoning traces.
  • Architecture combining a spatiotemporal video encoder with a structure-aware cardiac detector, providing spatially grounded anatomical cues alongside global motion dynamics.
  • Two-stage training pipeline (SFT → GRPO) with task-specific composite rewards unifying multi-label classification and report generation under a single RL framework.
  • State-of-the-art results on a private multi-view dataset and two public benchmarks (MIMICEchoQA, EchoNet-Dynamic).

Results

  • Private test set (multi-view): macro BAcc 67.4% — surpasses best external baseline (Lingshu, 50.3%) by 17.1%; macro F1 49.4% vs. best external (Chiron-o1, 23.9%). GRPO improves over SFT-only by +2.3% BAcc and +4.3% F1; per-class sign test \(p \approx 0.006\).
  • Low-prevalence conditions (private): AV stenosis (9.5% prevalence) — 82.2% BAcc vs. 54.7% strongest baseline; LV enlargement (5.7%) — 75.5% BAcc vs. 52.5%; bicuspid AV (2.5%) — 65.7% BAcc vs. 55.6%.
  • MIMICEchoQA (single-view): macro BAcc 59.0%, outperforming Chiron-o1 by 6.1%.
  • EchoNet-Dynamic (single-view): BAcc 60.2% for LV systolic dysfunction, +8.4% over Chiron-o1.
  • Report generation: GREEN clinical faithfulness score 0.800 (Tab. 3 in paper, not shown in excerpt).
  • Reasoning quality (LLM-as-judge, 1–5 scale): average 4.99/5 on five dimensions (Reasoning-Answer Agreement, Efficiency, Factual Correctness, Evidence Grounding, Terminology Accuracy) — matches SFT-only but far exceeds baselines (next best Lingshu: 4.45).
  • Cross-dataset consistency: average BAcc drop of 8.4% from private → MIMICEchoQA across five shared conditions; AV regurgitation most stable, LA enlargement shows largest drop.

Limitations

  • Public benchmarks (MIMICEchoQA, EchoNet-Dynamic) are single-view, preventing full evaluation of the multi-view architecture on public data.
  • Training relies on a private dataset and bounding-box annotations (2,689 studies) for the structure detector, limiting reproducibility.
  • Reasoning trace quality is evaluated via LLM-as-a-judge (Mistral-7B) rather than expert clinician review.
  • Cross-dataset robustness shows notable degradation (8.4% BAcc drop) when moving from multi-view private to single-view public data.
  • Only one class (LV systolic dysfunction) evaluated on EchoNet-Dynamic; no multi-label or report generation results on that benchmark.

Relevance to Vision-Language Models

This work demonstrates how VLMs can be extended from single-image QA to multi-view temporal reasoning in a clinical setting, combining spatiotemporal video encoding with structured anatomical grounding via object detection. The two-stage SFT+GRPO pipeline offers a concrete recipe for aligning VLM outputs with domain-specific correctness (via composite reward design) in a low-data setting, and the focus on explicit reasoning traces addresses a key barrier to clinician trust in medical VLMs. It connects to the broader trend of RL-based reasoning alignment in VLMs (cf. MedVLM R1, Chiron-o1).