Skip to content

EXACT: an explainable anomaly-aware vision foundation model for analysis of 3D chest CT

🕒 Published (v1): 2026-04-27 07:57 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EXACT is a 3D chest CT foundation model that replaces global contrastive (CLIP-style) embeddings with intrinsic voxel-level anomaly-aware maps (AAmaps), produced via a Y-shaped Mamba dual-decoder architecture jointly trained on organ segmentation and multi-instance anomaly detection. Pre-trained on 25,692 CT-report pairs without manual voxel annotations, it achieves state-of-the-art performance across multi-disease diagnosis, zero-shot anomaly localization, supervised segmentation, and radiology report generation on multinational cohorts. An EXACT-CHAT extension grounds LLM-generated reports to spatial voxel evidence.

Problem

Existing 3D chest CT foundation models (CT-CLIP, fVLM, MedVista3D, T3D, SimCroP) compress volumetric features into compact 1D embeddings via global average pooling or [CLS] tokens, irreversibly collapsing voxel-wise spatial structure. This prevents precise localization of focal/sparse pathologies, forces interpretability to rely on post-hoc gradient maps (Grad-CAM) that are structurally ambiguous, and causes severe performance degradation under distribution shift. No existing 3D CT FM achieves intrinsic voxel-level anomaly localization without dense supervision.

Method

EXACT uses a Y-Mamba backbone: a shared Mamba-based encoder branches into (1) an organ segmentation decoder and (2) a multi-instance learning (MIL) anomaly detection decoder that outputs 18 disease-specific AAmaps at voxel resolution. Pre-training is fully weakly supervised on CT-RATE (25,692 non-contrast CT/report pairs): organ masks are derived automatically via Segment Anything by Text; disease pseudo-labels are extracted from reports via RadBERT. The anatomy-constrained MIL objective treats each organ region as an instance bag, confining anomaly evidence to anatomically plausible tissue. At inference, AAmaps are element-wise masked by organ segmentations and aggregated for zero-shot diagnosis or localization. EXACT-CHAT freezes the EXACT encoder and integrates it with LLaMA-3.1-8B-Instruct via a LLaVA-style projector, with structured AAmap-derived diagnostic priors passed as text tokens; an optional GPT-4.1 refinement step corrects hallucinations and missed findings.

Key Contributions

  • Y-Mamba dual-decoder architecture producing dense, spatially explicit AAmaps without any manual voxel-level annotations
  • First 3D chest CT FM to achieve intrinsic (not post-hoc) voxel-level zero-shot anomaly localization via anatomy-constrained MIL
  • Anatomy-aware weak supervision paradigm as a clinically grounded alternative to CLIP for volumetric medical imaging
  • EXACT-CHAT: spatially grounded report generation that links narrative findings to voxel-level evidence, reducing hallucinations
  • Comprehensive multinational, multi-center benchmark (CT-RATE / RAD-ChestCT / MianYang / COVID-19 / MosMed) covering five clinically motivated tasks

Results

Multi-disease diagnosis (18 abnormalities, zero-shot): - CT-RATE (internal): AUROC 0.830 vs. best zero-shot baseline fVLM 0.778; outperforms fine-tuned T3D (0.802) - RAD-ChestCT (OoD): AUROC 0.728 vs. MedVista3D 0.710; all CLIP baselines ≤ 0.720 even with fine-tuning - MianYang (OoD): AUROC 0.758 vs. fVLM 0.716; pericardial effusion AUROC 0.84 vs. CT-CLIP 0.63

Zero-shot anomaly localization: - COVID-19 (OoD): DSC 0.435 vs. BiomedParse-v2 0.340; Hit Rate@5% 0.950 vs. 0.550 - MosMed (OoD): DSC 0.363 vs. BiomedParse-v2 0.254; CT-CLIP near-zero DSCs (0.004–0.023) across all datasets - Zero-shot DSC of 0.435 on COVID-19 approaches fine-tuned EXACT-Seg DSC 0.476

Supervised segmentation (EXACT-Seg): - COVID-19: DSC 0.476 vs. SegMamba 0.332 (+43.4%, P < 0.001) - MosMed: DSC 0.454 vs. SegMamba 0.352 (P < 0.001)

Report generation (RadBERT-F1, clinical efficacy): - CT-RATE: EXACT-CHAT (Refined) 0.501 vs. CT-CHAT 0.305, CT-GRAPH 0.296 - RAD-ChestCT: 0.441 vs. Hulu-Med 0.279 - MianYang: 0.410 vs. Hulu-Med 0.175; CT-CHAT collapsed to 0.073

Limitations

  • RadBERT label extraction requires manual pre-definition of 18 target abnormalities and disease-to-organ mappings, limiting scalability to rare or undefined pathologies
  • Dependency on a fine-tuned RadBERT for pseudo-label generation; more capable general-purpose LLMs could improve label quality
  • All evaluations are retrospective; prospective clinical validation assessing impact on radiologist accuracy, workflow efficiency, and patient outcomes is absent
  • Zero-shot DSC values are inherently modest on challenging internal datasets (ReX-Train DSC 0.050, ReX-Val DSC 0.071) due to difficulty of unsupervised 3D localization
  • COVID-19 external localization improvements did not reach statistical significance (P > 0.05) due to small sample size (n = 20)

Relevance to Foundation Models in Medicine

EXACT directly challenges the dominant paradigm of transplanting CLIP-style contrastive learning to volumetric medical imaging, demonstrating that task-specific inductive biases (anatomy-constrained MIL, dual-decoder spatial decoding) yield representations that are both more accurate and intrinsically interpretable — a key clinical trustworthiness requirement. The anatomy-aware weak supervision approach generalizes across diagnosis, localization, segmentation, and report generation from a single pre-training run, advancing the "one model, many tasks" goal for 3D medical AI. For researchers tracking foundation models in medicine, EXACT's results quantify the cost of global embedding compression under distribution shift and establish voxel-level grounding as an essential property for clinically deployable CT foundation models.