Fine-Grained Vision-Language Pretraining with Organ-Conditioned Pattern Tokens for CT Understanding¶

🕒 Published (v1): 2026-07-15 14:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

OCP-CT introduces organ-conditioned pattern tokens for CT vision-language pretraining by routing organ image/text tokens through private sparse Mixture-of-Experts and compressing them into fixed-size pattern tokens via learnable slot queries. Structured soft contrastive targets relax hard-negative bias across radiologically similar samples. On zero-shot abnormality diagnosis, OCP-CT achieves 84.5% AUROC on CT-RATE (+6.7 pp over the prior anatomy-aware state-of-the-art) and 69.9% on external RAD-ChestCT.

Problem¶

Global scan-report contrastive learning (CLIP-style) compresses all organ evidence into a single embedding, obscuring heterogeneous findings. Organ-level local alignment is also too coarse: the same lung can contain normal parenchyma, diffuse opacity, focal nodules, atelectasis, and pleural fluid simultaneously. No prior CT VLP method decomposes evidence at the level of organ-conditioned radiological patterns, and standard pairwise contrastive objectives treat all off-diagonal (non-paired) organ samples as uniformly hard negatives, penalizing cases that share related radiological patterns.

Method¶

OCP-CT adds an organ pattern pathway alongside a stable global CT-report contrastive branch (symmetric InfoNCE over global tokens $z^g_i$, $u^g_i$).

Organ token selection. Downsampled organ masks extract image tokens $V_{i,o}$ from a ResNet-18 3D feature map; a BioMedVLP CXR-BERT text encoder supplies organ-specific text evidence $T_{i,o}$ parsed from report fields.

Sparse MoE routing. Private image and text pattern expert banks ($M=4$ experts, top-$k=2$ routing) process each organ token independently: $$\tilde{v} = \sum_{m \in K^I(v)} \frac{p^I_m(v)}{\sum_{\ell \in K^I(v)} p^I_\ell(v)} \, PE^I_m(v).$$ A load-balancing auxiliary loss prevents expert collapse.

Slot pooling. Learnable slot queries $Q_o \in \mathbb{R}^{K_s \times C}$ ($K_s=6$) attend over routed tokens via cross-attention to produce $K_s$ continuous pattern tokens per organ per modality: $$S^I_{i,o} = A^I_{i,o}\,\tilde{V}_{i,o}, \quad A^I_{i,o} = \text{softmax}\!\left(\frac{(Q_o W^I_q)(\tilde{V}_{i,o} W^I_k)^\top}{\sqrt{d}}\right).$$ A slot diversity penalty $\mathcal{L}_\text{div}$ discourages slots from copying each other.

Paired pattern-token contrast. Within-organ pattern matching scores aligned pairs slot-by-slot: $$p^{(o)}_{ij} = \frac{1}{K_s}\sum_{k=1}^{K_s} \left\langle \frac{S^I_{i,o,k}}{\|S^I_{i,o,k}\|}, \frac{S^T_{j,o,k}}{\|S^T_{j,o,k}\|} \right\rangle.$$ These scores form a batch similarity matrix $P^{(o)}$, trained against structured soft targets $T^{(o)} = \lambda_d I + \lambda_a A_\text{abn} + \lambda_c A_\text{concept} + \lambda_l A_\text{loc} + \lambda_r A_\text{attr}$, built from RadGraph-XL/LLM-parsed report fields ($\lambda_d{=}1.0, \lambda_a{=}0.8, \lambda_c{=}0.8, \lambda_l{=}0.20, \lambda_r{=}0.15$). Pathology prompt anchoring and organ classification auxiliary heads supplement supervision.

Key Contributions¶

Formulates the organ-conditioned radiological pattern as the fundamental alignment unit for CT VLP, rather than the organ itself.
OCP-CT framework: private sparse MoE + organ slot queries producing paired continuous pattern tokens for cross-modal alignment.
Structured soft contrastive targets that weight off-diagonal batch samples by clinical similarity (abnormality labels, concept overlap, location/attribute agreement), mitigating hard-negative bias.
Gated query residual on the global branch that preserves average-pooled CT representation at initialization while enabling bounded adaptive readout.
Demonstrated zero-shot gains on two public CT benchmarks with organ-group and ablation analysis supporting each module.

Results¶

Zero-shot abnormality diagnosis (AUROC): - CT-RATE: OCP-CT 84.5% vs. fVLM 77.8% (+6.7 pp), COLIPRI-C 76.3%, CT-CLIP 73.1% - RAD-ChestCT (external): OCP-CT 69.9% vs. fVLM 68.0% (+1.9 pp), COLIPRI-C 69.1% (+0.8 pp) - CT-RATE additional metrics: ACC 78.1, F1 80.3, Precision 44.5 (vs. fVLM: 71.8, 75.1, 37.9)

Image–image retrieval (CT-RATE MAP@5/10/50): - OCP-CT 70.7 / 61.0 / 53.5 vs. best prior CT-CLIP/VocabFine 68.3 / 57.2 / 48.9 (+2.4/+3.8/+4.6)

Report–image retrieval (CT-RATE Recall@5/10/50/100): - OCP-CT 7.4 / 12.4 / 34.3 / 48.6 vs. CT-CLIP 2.9 / 5.0 / 18.0 / 28.7 (+4.5/+7.4/+16.3/+19.9)

Organ-group AUROC gains (vs. global-only baseline): lung +8.9, esophagus +8.0, heart +5.4, aorta +3.7, other +9.6 pp.

Limitations¶

Active organ set is restricted to four anatomies: lung, heart, esophagus, aorta; other lesion types (Medical material, Lymphadenopathy) fall outside conditioned routing.
Organ mask availability is required during pretraining; the paper notes empty masks are skipped but does not quantify how often this occurs.
Image encoder is ResNet-18, a relatively lightweight 3D backbone; scalability to larger volumetric encoders is not evaluated.
Report parsing relies on RadGraph-XL and LLM normalization for soft-target construction, introducing an external dependency and potential noise.
RAD-ChestCT gains over the prior best are marginal (+0.8–1.9 AUROC), suggesting domain generalization benefits are modest.
No evaluation on segmentation, grounding, or report generation; downstream scope is limited to diagnosis and retrieval.

Relevance to Vision-Language Models¶

OCP-CT directly addresses a core challenge in medical VLMs: the granularity mismatch between global vision-language contrastive objectives (CLIP-style) and the fine-grained, spatially localized evidence structures that clinicians actually reason over. The pattern-token bottleneck — combining sparse MoE routing with slot attention and structured soft targets — is a generalizable design principle for any domain where a single image encodes multiple heterogeneous semantic regions (e.g., scene understanding, satellite imagery). The structured soft target formulation is particularly relevant to the broader VLM literature on hard-negative sensitivity in contrastive learning. For researchers tracking VLMs in medical imaging, this paper establishes that sub-organ pattern decomposition, not just organ-level grounding, is necessary for closing the gap between global pretraining and fine-grained clinical understanding.