Skip to content

Health system learning achieves generalist neuroimaging models

🕒 Published (v1): 2025-11-23 22:34 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NeuroVFM is a volumetric visual foundation model trained on 5.24 million clinical MRI and CT volumes from a single health system using a novel self-supervised method called Vol-JEPA, without any human annotation or radiology report supervision. It achieves state-of-the-art performance on neuroimaging diagnosis (AUROC ~92.5–92.7%) and outperforms GPT-5 and Claude Sonnet 4.5 on radiology report generation and clinical triage when paired with an open-source LLM. The work formalizes "health system learning" as a scalable paradigm for building generalist medical foundation models from uncurated, real-world clinical data.

Problem

Frontier AI models (GPT-5, DINOv3) are trained on internet-scale public data, which severely underrepresents clinical neuroimaging due to privacy constraints on MRI/CT (facial identifiability). Prior medical models rely on data curation, human annotations, or radiology report supervision, limiting scale and generalizability. This leaves a gap between publicly-trained frontier models and the performance needed for safe clinical deployment.

Method

Vol-JEPA (Volumetric Joint-Embedding Predictive Architecture): A 3D extension of I-JEPA/V-JEPA applied to volumetric neuroimages. A volume is tokenized into 4×16×16 voxel patches; ~85% of tokens are masked using a neuroanatomy-informed strategy (foreground-focused, context from peripheral structures predicting deep structures, or vice versa). A student 3D Vision Transformer encodes visible context patches; a predictor network reconstructs masked target patch representations using positional encodings. A teacher encoder (EMA-updated, no gradients) provides target representations; training minimizes smooth L1 loss in latent space. This requires no voxel-level augmentations, negative pairs, or radiology reports, enabling scaling to the full uncurated UM-NeuroImages dataset (566,915 studies, 5.24M volumes, 20 years of acquisitions at Michigan Medicine). For downstream report generation, frozen NeuroVFM tokens are connected to Qwen3-14B via LLaVA-1.5-style visual instruction tuning.

Key Contributions

  • Health system learning paradigm: Formalized as five principles for training generalist medical AI directly from uncurated routine clinical data.
  • UM-NeuroImages dataset: 566,915 studies / 5.24M 3D volumes (MRI + CT), an order of magnitude larger than prior 3D neuroimaging datasets, with a clinically-validated diagnostic ontology of 74 MRI and 82 CT diagnoses auto-labeled via GPT-4.1-mini.
  • Vol-JEPA: Scalable volumetric JEPA with neuroanatomy-informed masking; first application of JEPA to volumetric medical images.
  • Emergent neuroanatomic understanding: Unsupervised organization of tokens into a neuroanatomic manifold, zero-shot cross-modal transfer (CT↔MRI) without additional training, and zero-shot semantic anatomic matching across modalities and protocols.
  • Diagnostic grounding: A novel "classify-then-aggregate" AB-MIL variant that yields class-specific, directionally interpretable attention maps without region-level annotations.
  • Report generation superiority: NeuroVFM-LLaVA outperforms GPT-5 and Claude Sonnet 4.5 on triage accuracy (93.5% vs. 83.0% vs. 71.5%), urgent finding detection, all NLP metrics (METEOR, ROUGE), and expert preference (>2:1 over GPT-5).

Results

  • Diagnostic classification (UM-NeuroBench): Mean AUROC 92.7 ± 0.4% (82 CT tasks) and 92.5 ± 0.4% (74 MRI tasks), outperforming HLIP, DINOv3, and BiomedCLIP.
  • External benchmarks: Outperformed internet-scale baselines by wide margins on ADNI/AIBL (Alzheimer's), PPMI (Parkinson's), ABIDE (autism), CQ500 (structural), RSNA (ICH subtypes).
  • Scaling laws: Log-linear relationship between positive training examples and F1 score (F1 = 0.267 + 0.120·log₁₀(n)), holding over 4 orders of magnitude across modalities and architectures.
  • Triage accuracy (300-study expert set): NeuroVFM 93.5% vs. GPT-5 83.0% vs. Sonnet 4.5 71.5%.
  • Report errors vs. GPT-5: NeuroVFM key-finding error rate ~10% vs. ~20% for GPT-5; >2× fewer hallucinations; >3× fewer laterality errors.
  • Expert preference: NeuroVFM preferred >2:1 over GPT-5 by three blinded clinical experts (Fleiss' κ = 0.718).
  • Cross-modal transfer: MRI-trained Chiari malformation classifier transfers to CT with ΔAUROC +16.1% vs. DINOv3.

Limitations

  • Single health system (University of Michigan) for pretraining; potential site-specific distribution shift not fully characterized.
  • Prospective clinical validation in live workflows not yet performed; evaluation is retrospective.
  • No integration of longitudinal data, pathology, genomics, or other multimodal clinical streams (lab values, clinical notes beyond indications).
  • Report generation evaluation used only 300 expert-selected studies; broader prospective triage validation is needed.
  • Vol-JEPA training and UM-NeuroImages are not publicly released (model weights available, dataset restricted by privacy).
  • Interpretability via AB-MIL provides attention maps but not formal uncertainty quantification; not yet validated in human-AI collaborative interfaces.

Relevance to Foundation Models in Medicine

NeuroVFM directly advances the central question in medical foundation models: whether private, high-volume clinical data can substitute for internet-scale pretraining to achieve generalizable, clinically safe performance. The "health system learning" paradigm is a principled, reproducible framework (five explicit principles) that complements internet-scale training and positions domain-specific VFMs as plug-in modules within agentic systems — a concrete architectural path for deploying medical FM components alongside GPT/Claude-class reasoning models. The Vol-JEPA architecture and empirical scaling laws provide transferable methodology applicable beyond neuroimaging to any organ system with sufficient institutional-scale PACS data. The hallucination-reduction results in report generation are particularly significant for the clinical safety dimension of medical FM deployment.