Skip to content

Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification

🕒 Published (v1): 2025-12-15 00:01 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AnyMC3D adapts frozen 2D foundation models (FMs) to 3D medical image classification by adding ~1M lightweight, task-specific LoRA adapters and a query-based slice aggregation module per task. The paper identifies three critical pitfalls in prior FM benchmarking (data-regime bias, suboptimal adaptation, insufficient task coverage), establishes a 12-task diverse benchmark, and shows that properly adapted 2D FMs—including general-purpose ones like DINOv2/v3—outperform 3D-native medical FMs. AnyMC3D achieves 0.894 mean AUC across 10 tasks and won 1st place in the VLM3D challenge.

Problem

Prior 3D medical FM research suffers from three intertwined flaws: (1) evaluation is biased toward low-data/few-shot regimes where clinical performance remains unacceptably low; (2) adaptation strategies are suboptimal—frozen backbone + average/median pooling systematically underestimates FM potential (ΔAUC=0.11 on CT-RATE for the same backbone); and (3) task coverage is too narrow (single modality or anatomy), preventing assessment of true generalizability. The dominant one-model-per-task paradigm also scales poorly to new clinical applications.

Method

AnyMC3D keeps a single 2D ViT backbone (DaViT-365M, SigLIP-432M, or ViT-L-300M) frozen and attaches task-specific plugins (~1M params each):

  • In-plane encoding: LoRA adapters (rank=8, α=16) applied to patch embedding and all self-attention projections; each 3D slice is encoded independently to produce a CLS-token embedding.
  • Slice aggregation: A learnable task query \(\mathbf{q}_t\) performs query-based attention pooling over the stack of slice embeddings in a permutation-invariant manner (no ordered RNN/Transformer), producing a volume embedding fed to a classification head.
  • Multi-view extension: View-specific LoRA adapters and queries encode each MRI view/sequence separately; a second attention pooling fuses view embeddings for patient-level prediction.
  • Auxiliary pixel-level supervision: Patch tokens from the last ViT block are reshaped into a pseudo-3D feature volume and decoded by a lightweight 3D decoder to pixel-wise logits; segmentation loss is added during training on the subset of labeled volumes, then removed at inference.
  • Interpretable heatmaps: Per-slice class-to-patch attention maps (averaged over heads, upsampled) are weighted by slice importance scores from the task query and stacked to form a 3D saliency map.

Key Contributions

  • Formal identification of three critical pitfalls (data-regime bias, suboptimal adaptation, insufficient task coverage) prevalent in medical FM evaluation.
  • AnyMC3D: a scalable single-backbone framework adapting 2D FMs to arbitrary 3D classification tasks with ~1M trainable parameters per task.
  • A 12-task benchmark spanning CT and MRI, multiple anatomies (head, chest, abdomen, shoulder), and realistic dataset sizes (1,140–50,188 volumes), including public (RATIC, PANORAMA, CT-RATE) and private datasets.
  • Empirical demonstration that general-purpose 2D FMs (DINOv2/v3) match medical-specific 2D FMs (MII, MedGemma) when properly adapted, and both surpass 3D medical FMs.
  • 1st place in the VLM3D challenge with ~0.5M trainable parameters against 118 participants using large-scale CT pretraining.

Results

  • Average AUC across 10 tasks: AnyMC3D (MII) = 0.894 (rank 1.7); AnyMC3D (DINOv3) = 0.894 (rank 2.0) vs. best prior method MST = 0.869 (rank 4.0), VoCo = 0.793 (rank 7.0), MedicalNet = 0.783.
  • Trainable parameters: 1.20–2.01M (AnyMC3D) vs. 23.05M (MST), 50.49M (VoCo)—10–50Ă— fewer.
  • Adaptation impact: MII with naive linear probing = 0.785 AUC; with AnyMC3D = 0.894 (+0.109); MedGemma from 0.690 to 0.866.
  • PDAC detection (T5): AnyMC3D (DINOv3) = 0.962 AUC vs. PANORAMA challenge winner PanDx = 0.949; adding auxiliary segmentation supervision raises it to 0.973.
  • CT-RATE chest (T11): AnyMC3D (DINOv2) mean AUC = 0.884 vs. CT-Net = 0.631, CT-CLIP = 0.748.
  • Head CT emergency triage (T12): AnyMC3D (DINOv2) mean AUC = 0.820 vs. DeepCNTD = 0.768 across 75 findings.
  • Data efficiency: With 20% training data, AnyMC3D (DINOv3) achieves 0.924 AUC on T3 vs. 3D DenseNet's 0.741; matches DenseNet trained on 60% data (3Ă— efficiency).
  • DINOv2 vs. DINOv3: Negligible difference after adaptation (Tab. 8).

Limitations

  • Findings are based only on the FMs evaluated (DINOv2/v3, MII, MedGemma); different backbones may exhibit different adaptation behavior.
  • All tasks use in-domain CT/MRI; generalization to out-of-domain modalities (e.g., PET) is untested.
  • Auxiliary pixel-level supervision requires expensive segmentation labels; weaker supervision (bounding boxes, radiology reports) is unexplored.
  • DINOv3's improved spatial features (Gram anchoring) do not transfer to 3D classification benefit; potential for dense prediction tasks (segmentation) is unverified.
  • Several tasks use private datasets, limiting full reproducibility.
  • The permutation-invariant slice aggregation ignores through-plane spatial ordering, which may be suboptimal for tasks requiring precise 3D localization.

Relevance to Foundation Models in Medicine

This paper directly challenges the prevailing assumption that 3D-native medical FMs are necessary for volumetric tasks, demonstrating that efficient 2D FM adaptation is both more parameter-efficient and empirically superior—a paradigm shift for how the community should develop and evaluate medical FMs. The identification of data-regime bias as a systematic benchmarking flaw is methodologically important: many prior claims about medical FM superiority may be artifacts of few-shot evaluation, not genuine generalization. The finding that general-purpose FMs (DINOv2/v3) match domain-specific medical FMs with proper adaptation challenges the ROI of expensive medical-specific pretraining, with implications for resource allocation in FM development. AnyMC3D's scalable plugin architecture—one frozen backbone, lightweight per-task LoRA + query—offers a practical blueprint for deploying FMs across diverse clinical classification tasks without retraining entire models.