Skip to content

Structure is Supervision: Multiview Masked Autoencoders for Radiology

🕒 Published (v1): 2025-11-27 10:20 UTC · Source: Arxiv · Venue: TMLR · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MVMAE is a self-supervised pretraining framework for chest radiography that treats the multi-view structure of radiology studies (frontal, lateral projections) as free supervision by combining per-view masked image reconstruction with cross-view alignment. MVMAE-V2T extends this with a vision-to-text captioning objective that uses radiology reports only during pretraining, keeping inference vision-only. Both variants consistently outperform supervised and vision-language baselines on disease classification across three large public datasets.

Problem

Existing self-supervised methods for chest X-rays treat images from the same study as independent instances, discarding the geometric and semantic coherence between co-acquired projections (e.g., frontal and lateral views). Reconstruction-only MAEs are view-specific; contrastive methods ignore fine-grained reconstruction; VLMs depend on report availability and emphasize semantic rather than geometric correspondence. No prior method couples masked reconstruction with cross-view regularization at scale.

Method

MVMAE uses a ViT-Base encoder with a lightweight decoder, applying a high masking ratio (90%) independently to each view within a study. The joint objective has two terms: (1) LRec — standard MAE pixel reconstruction over masked patches per view; (2) LAlign — pairwise MSE between visible encoder tokens of different views within the same study, weighted by β with a beta-annealing schedule. Learnable modality embeddings (frontal/lateral/unknown) condition the encoder on projection type without per-view heads. MVMAE-V2T adds a third term (3) LCE — a causal language modeling loss where a transformer decoder autoregressively generates the study's radiology report conditioned on unmasked visual embeddings via cross-attention, using γ weighting. Pretraining is performed jointly on four datasets (MIMIC-CXR, CheXpert Plus, PadChest, Chest X-ray; ~870K+ images) using a study-centric FLU (Frontal–Lateral–Unknown) policy that retains all studies with ≥1 valid projection.

Key Contributions

  • First multi-view masked autoencoder for radiology combining per-view reconstruction with cross-view alignment as a self-supervisory signal.
  • MVMAE-V2T: reports used purely as auxiliary pretraining signal; vision-only at inference, avoiding dependence on report availability downstream.
  • Clinically realistic study-centric evaluation policy with analysis of performance scaling with number of available views.
  • Unified benchmark integrating MIMIC-CXR, CheXpert Plus, and PadChest under a harmonized 14-label CheXpert ontology.

Results

  • Full-data combined AUROC (14 labels): MVMAE 87.41 > MVMAE-V2T 86.74 > Independent 86.60 > CheXagent (ViT-L/14, linear probe) 86.25 > Supervised 84.83.
  • Top-5 labels combined AUROC: MVMAE 91.78 > MVMAE-V2T 91.65 > CheXagent 90.64 > Independent 88.99 > Supervised 86.16.
  • Per-dataset: MVMAE achieves highest AUROC on MIMIC-CXR (82.46), CheXpert (83.24), PadChest (89.17), and Combined (87.41).
  • Label efficiency: ~80% AUROC requires 5K labeled samples for MVMAE variants vs. 10K for Independent vs. 50K for Supervised baseline.
  • Calibration (Brier score, linear probe, 20K samples): MVMAE best among pretraining baselines on MIMIC-CXR (0.0903) and Combined (0.0835); supervised baseline still lowest overall.
  • Multi-view benefit: MVMAE consistently outperforms Independent variant in both single-view and two-view evaluation settings.

Limitations

  • Evaluation restricted to chest radiography; generalizability to other modalities (MRI, CT) is proposed but not demonstrated.
  • MVMAE-V2T advantage over MVMAE diminishes as labeled data increases; text supervision primarily benefits low-label regimes.
  • Calibration gains over baselines are modest and MVMAE-V2T is notably worse than MVMAE on Brier score (e.g., Combined 0.0905 vs. 0.0835).
  • All datasets are chest X-rays from critical-care settings; performance on outpatient or pediatric protocols is unknown.
  • CheXagent is evaluated in linear probing while MVMAE uses full finetuning, making direct comparisons in Table 1 not strictly controlled.

Relevance to Foundation Models in Medicine

MVMAE exemplifies a key design principle for medical foundation models: exploiting domain-specific data structure (multi-view study organization) as a free, scalable supervisory signal rather than relying on expensive labels or general-purpose internet pretraining. Its label-efficiency results (10× fewer labels to reach comparable performance vs. supervised) directly address the chronic annotation bottleneck in clinical AI. The MVMAE-V2T design—using reports as pretraining-only auxiliary supervision to preserve vision-only inference—is a practically important pattern for deploying models in institutions where structured reports are available during training but not guaranteed at runtime. This work advances the understanding of how to adapt SSL paradigms (MAE + contrastive alignment) to the structural realities of clinical imaging, contributing a reusable blueprint for modalities beyond CXR.