Skip to content

CardioBench: Do Echocardiography Foundation Models Generalize Beyond the Lab?

🕒 Published (v1): 2025-10-01 05:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CardioBench is a standardized benchmark unifying eight public echocardiography datasets across nine clinical tasks (four regression, five classification) to evaluate foundation models under zero-shot, linear probing, and retrieval settings. It reveals that no single model dominates: cardiac-specific models lead on functional regression, general-purpose encoders (SigLIP2, DINOv3) are surprisingly competitive on several tasks, and retrieval-based approaches generalize more consistently. The benchmark provides public preprocessing, data splits, and evaluation scripts to enable reproducible comparison.

Problem

Echocardiography foundation models are predominantly trained and evaluated on private datasets, making fair cross-model comparison impossible. The only prior public benchmark (ETAB) predates modern multimodal foundation models and omits zero-shot and cross-modal alignment evaluation. Echocardiography also presents unique challenges—high frame-to-frame cosine similarity, noisy acquisitions, multi-view anatomy—that are not addressed by existing benchmarks.

Method

CardioBench assembles eight public echocardiography datasets (EchoNet-Dynamic, CAMUS, EchoNet-Pediatric, EchoNet-LVH, CardiacNet, HMC-QU, TMED-2, segRWMA) into a unified evaluation suite. Tasks span LV ejection fraction regression (MAE), structural regression of cardiac wall dimensions (LVIDd, IVSd, LVPWd), disease classification (ASD, PAH, STEMI, aortic stenosis), and view recognition (A2C, A3C, A4C). Evaluation protocols are: (1) zero-shot — inference using only components released by original authors, including text-prompt alignment for contrastive models; (2) linear probing — a frozen-encoder linear head trained on target dataset labels; (3) kNN probing for view classification; and (4) cross-modal alignment analysis — projecting visual embeddings onto a text-derived EF axis and computing Pearson correlation. Demographic subgroup analyses stratify EF error by sex, age, BMI, and image quality. Models evaluated: EchoCLIP, EchoPrime, PanEcho, EchoFM (probing only), BioMedCLIP, DINOv3, SigLIP2.

Key Contributions

  • First comprehensive public benchmark for echocardiography foundation models, covering 9 tasks across 8 datasets with standardized splits, preprocessing, and evaluation scripts.
  • Systematic zero-shot, linear probing, and cross-modal alignment comparison of cardiac-specific, biomedical, and general-purpose encoders under identical protocols.
  • Quantitative demonstration that temporal modeling (PanEcho) is essential for functional regression and retrieval-based matching (EchoPrime) offers complementary robustness.
  • Cross-modal alignment analysis showing EchoCLIP is the only model to encode a physiologically meaningful EF axis (r=0.52 on EchoNet-Dynamic); general-purpose models achieve near-zero alignment despite competitive MAE, suggesting artifact sensitivity rather than clinical understanding.
  • Demographic subgroup analysis revealing that PanEcho and EchoPrime have larger bias across sex, age, and BMI strata, while SigLIP2/DINOv3 maintain more uniform error distributions.

Results

Zero-shot regression (LV EF, MAE ↓): - PanEcho best on EchoNet-Dynamic (5.79) and EchoNet-Pediatric (9.10); EchoPrime second on Dynamic (7.78) - SigLIP2 competitive on CAMUS (9.28), outperforming most cardiac-specific models

Zero-shot structural regression (EchoNet-LVH, MAE ↓): - EchoPrime best on LVIDd (0.36), IVSd (0.21), LVPWd (0.18); PanEcho and general-purpose models comparable

Zero-shot classification (F1-macro ↑): - EchoCLIP leads on ASD (47.88%), PAH (46.96%), STEMI (52.51%, +18.3% over next best) - PanEcho leads on aortic stenosis (TMED-2: 58.90%) - SigLIP2/DINOv3 top RWMA view detection on A2C, A3C, A4C

Linear probing: - SigLIP2 surpasses all models on ASD (68.49% F1, +32% over zero-shot), leading specialized models by ~10% - EchoPrime best on PAH (63.36%) and STEMI (80.00%) - PanEcho maintains advantage on structural regression (LVIDd MAE 0.35) with minimal probing gain (Δ≤0.03)

View classification (zero-shot, F1-macro): - EchoPrime leads 5/8 datasets via its supervised view classifier head - BioMedCLIP best on CardiacNet (76.11%) and TMED-2 (62.86%); SigLIP2 best on CAMUS (57.01%) - DINOv3 collapses to near-zero F1 on several datasets

Limitations

  • EchoFM cannot perform zero-shot evaluation (no text encoder; task-specific heads not released), limiting its comparability.
  • EchoPrime's private training database could inflate retrieval-based results via test-train similarity leakage.
  • Eight datasets still represent limited acquisition diversity; benchmark cannot capture all real-world distribution shifts.
  • Zero-shot evaluation for contrastive models depends heavily on prompt design, introducing sensitivity to prompt engineering choices not fully ablated.
  • SigLIP2's lower MAE on poor-quality CAMUS scans (vs. good-quality) suggests artifact sensitivity rather than cardiac understanding—a failure mode not captured by aggregate metrics alone.
  • Linear probing and kNN probing exclude full fine-tuning, which may underestimate the ceiling of cardiac-specific models on small datasets.

Relevance to Foundation Models in Medicine

CardioBench directly addresses a critical reproducibility gap in medical foundation model evaluation: the field's over-reliance on private datasets has made it impossible to determine whether cardiac-specific pretraining genuinely outperforms general-purpose encoders. The finding that SigLIP2 and DINOv3 match or surpass domain-specific models on several tasks—while lacking meaningful clinical alignment—raises fundamental questions about what "foundation model capability" means in specialized medical imaging and challenges the assumption that domain-specific training is always necessary. The cross-modal alignment methodology (text-axis Pearson correlation) is a transferable evaluation technique for other medical imaging domains where continuous physiological variables matter. The benchmark's architecture-agnostic, multi-protocol design sets a methodological template for similar benchmarks in cardiology MRI, pulmonology, or ophthalmology.