Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography¶

🕒 Published (v1): 2026-06-24 08:24 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; 3D CT VLP with hybrid encoding; directly buildable medical foundation recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CT-DiagVLM introduces a VLP framework for 3D CT that replaces ViT patch embedding with a 3D CNN backbone (CNN–ViT hybrid) and adds disease-level contrastive learning via learnable query tokens. It achieves state-of-the-art zero-shot multi-disease diagnosis on CT-RATE (84.4% AUC, +5.1%) and RadChestCT (75.4% AUC, +5.4%), with a +9.8% AUC gain on a demanding 60-disease benchmark.

Problem¶

Existing 3D CT VLP methods suffer from two compounding failures: (1) visual backbones are ill-suited to volumetric data — naive 3D ViT extensions are computationally prohibitive and lose spatial detail, while 3D CNNs are incompatible with pre-trained cross-modal priors; (2) alignment is coarse — global image–report contrastive learning and even organ-level methods conflate distinct co-occurring pathologies (e.g., "atelectasis" vs. "emphysema" vs. "lung nodules") under a generic "abnormal" label, destroying disease-discriminative signal.

Method¶

Three co-designed components:

CNN–ViT Hybrid Encoder (MSF-PE). A 3D ResNet-18 extracts a multi-scale CNN hierarchy $\mathcal{F}_{CNN} = \{F_1, F_2, F_3, F_4\}$. Each stage is projected to a common channel dimension $d$, downsampled to $F_4$'s resolution, concatenated, and fused via 3D convolution into $F_{fused}$. This serves as patch embeddings for a ViT initialized from pre-trained SigLIP-2 weights, injecting local anatomical priors while preserving cross-modal pre-training compatibility.

Disease-Level Contrastive Learning. Per organ $k$, a set of learnable condition query vectors $\{q_k^{(c)}\}_{c \in \mathcal{C}_k}$ (one per disease + healthy state) extracts condition-specific textual embeddings from the full organ report via cross-attention: $$s_{b,k}^{(c)} = \text{CrossAttn}(q_k^{(c)}, T_{b,k}, T_{b,k})$$ For each condition $c$, positive textual embeddings across the batch form a set $S_k^{(c)}$, and organ-level visual features $v_{b,k}$ are pulled toward same-disease embeddings and pushed from others via a batch-wise InfoNCE loss $\mathcal{L}_{disease}$. Labels come from ground-truth or LLM-extracted pseudo-labels (Qwen3-Max).

Diagnosis-Aware Prompt Strategy. Rather than hand-crafted binary templates, inference prototypes $p_k^{(c)}$ are built by averaging query-guided embeddings over $M \geq 500$ training reports per condition–organ pair. Zero-shot scores are cosine similarities between visual embeddings and these prototypes, bridging the pre-training/inference distribution gap.

Total loss: $\mathcal{L}_{total} = \mathcal{L}_{dice} + \lambda_1 \mathcal{L}_{anat} + \lambda_2 \mathcal{L}_{disease}$ with $\lambda_1 = \lambda_2 = 0.5$.

Key Contributions¶

CNN–ViT hybrid encoder (MSF-PE): replaces ViT patch embedding with 3D ResNet-18 + multi-scale fusion, retaining SigLIP-2 pre-trained weight compatibility while capturing local anatomical detail.
Disease-level contrastive learning: learnable per-condition query tokens dynamically extract disease-specific semantics from unstructured reports without explicit report decomposition, enabling fine-grained disentanglement of co-occurring pathologies.
Diagnosis-aware prompt strategy: real clinical report phrases aggregated into per-condition prototypes replace hand-crafted templates, improving zero-shot diagnostic reliability.
LLM-augmented scalability: Qwen3-Max pseudo-labels extend alignment to 60 diseases while prototype averaging mitigates label noise.

Results¶

CT-RATE (internal, 16 diseases): 84.4% AUC vs. ViSD-Boost 79.0% (+5.4%), fVLM 77.8% (+6.6%), base model 79.3% (+5.1%).
RadChestCT (external, 16 diseases): 75.4% AUC vs. ViSD-Boost 69.4% (+6.0%), fVLM 68.0% (+7.4%).
CT-RATE 60-disease benchmark: 85.6% AUC vs. base model 75.8% (+9.8% AUC); ACC 85.3% vs. 73.8%.
Ablation (CT-RATE): CNN–ViT hybrid alone: +1.0% AUC over base; diagnosis-aware prompt: additional +2.1%; disease-level contrastive loss: additional +2.0%; combined: +5.1%.
Strong performance on truly unseen held-out diseases (strict zero-shot hold-out experiment, details in appendix).
Transferable to radiology report generation (evaluated with BLEU, METEOR, and diagnostic accuracy).

Limitations¶

Relies on TotalSegmentator-generated pseudo organ masks for U-Net supervision; errors in segmentation propagate to organ-level alignment.
Disease label set and query tokens are fixed at pre-training time; adding new diseases requires re-training or at minimum re-computing prototypes.
LLM pseudo-label quality (Qwen3-Max) is not verified against expert annotation; noise impact is mitigated by prototype averaging but not eliminated.
Evaluated only on chest CT; generalization to other anatomical regions (abdomen, brain) is not demonstrated.
The 60-disease pseudo-label experiment uses in-distribution CT-RATE data; external validation for the expanded label set is absent.

Relevance to Vision-Language Models¶

This work directly advances 3D medical VLM architecture design by showing how to graft a 3D CNN front-end onto a pre-trained 2D ViT (SigLIP-2) without discarding cross-modal priors — a transferable pattern for any volumetric modality. The disease-level contrastive learning via learnable query tokens is a fine-grained alignment technique that pushes beyond global image–text and organ-level CLIP-style objectives toward true entity-level disentanglement, relevant to VLM research on structured scene understanding. The diagnosis-aware prototype prompt strategy is a practical solution to the pre-training/inference gap that sidesteps prompt engineering by grounding zero-shot inference in retrieved clinical language. For researchers tracking VLMs, this paper provides a concrete, ablation-validated recipe for domain-adapted VLP where semantic granularity and architectural compatibility with foundation model weights are both critical.