Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge¶

🕒 Published (v1): 2026-07-12 23:09 UTC · Source: Arxiv · link

Why this paper was selected

Anatomy-hierarchical VLP from CT+reports; strong medical VLM pretraining recipe for organ-level grounding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

OKA-CT is a two-stage vision-language pretraining framework for chest CT that converts free-text radiology reports into organ-conditioned hierarchical knowledge and reuses that hierarchy to supervise both visual representation learning and structured contrastive alignment. By treating non-paired CT-report cases sharing organ-level findings as weak semantic positives rather than hard negatives, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 on CT-RATE and 72.2 on RAD-ChestCT, outperforming all prior CT VLP baselines.

Problem¶

Existing CT VLP methods align either whole scans with entire reports (global) or local image regions with text fragments (local), neither of which exploits the natural organ-hierarchical structure of radiology reports. Global methods compress multi-organ findings into a single embedding, diluting sparse but clinically critical observations. Local methods match regions to text fragments without using the internal report hierarchy as a supervisory signal. Additionally, standard contrastive learning treats all non-paired samples as hard negatives, introducing false-negative supervision when patients share the same organ-level abnormalities.

Method¶

OKA-CT operates in two stages, sharing a single 3D ResNet-18 feature lattice \(F_i = E_I(x_i)\).

Report Knowledge Extraction. Each free-text report \(r_i\) is parsed with RadGraph-XL to extract clinical entities/relations, then structured by Qwen3-8B into organ-conditioned slots \(z_i^o = \{a_i^o, c_i^o, \ell_i^o, h_i^o\}\) for each organ \(o \in \{\)lung, heart, esophagus, aorta\(\}\), encoding abnormality status, disease/finding concepts, anatomical location, and severity attributes.

Stage 1 (Visual Evidence Learning). The visual encoder is trained jointly on: (a) a query-based global branch that aggregates disease-relevant volume cues via learnable queries (\(g_i = W_g \,\text{Pool}(\text{Attn}(Q_g, F_i))\)), supervised by case-level disease labels with BCE loss; and (b) organ-conditioned branches that pool mask-selected features \(v_i^o = \text{Pool}(\{F_i(p) \mid m_i^o(p)=1\})\) and apply four organ-specific prediction heads (\(\psi_a^o, \psi_c^o, \psi_\ell^o, \psi_h^o\)) supervised by the extracted hierarchy slots. Stage 1 loss: \(\mathcal{L}^{(1)} = \mathcal{L}_\text{global} + \lambda_1 \mathcal{L}^{(1)}_\text{organ}\).

Stage 2 (Structured Contrastive Alignment). The visual encoder is frozen; projection and text-alignment components are trained. Global CT-report contrastive learning uses standard bidirectional InfoNCE. Organ-level alignment constructs a soft target matrix per organ:

\[T^o = I + \alpha S_a^o + \beta S_c^o + \gamma S_\ell^o + \sigma S_h^o\]

where \(S_f^o(i,j) = \mathbf{1}[\text{overlap}(f_i^o, f_j^o) > 0]\) marks non-paired cases sharing the same organ-level evidence in field \(f\). The organ loss uses a raw-mass weighted cross-entropy (not row-normalized targets) to preserve the count of reliable semantic neighbors. Stage 2 loss: \(\mathcal{L}^{(2)} = \mathcal{L}^{(2)}_\text{global} + \frac{\lambda_2}{|O|}\sum_o \mathcal{L}^{(2),o}_\text{organ}\).

Key Contributions¶

OKA-CT framework: two-stage CT-report VLP that reuses report-derived organ hierarchy for both visual representation learning and contrastive alignment.
Automated organ-hierarchical knowledge extraction: radiology report parser (RadGraph-XL) + LLM (Qwen3-8B) pipeline that produces structured organ slots (abnormality, concept, location, attribute) without manual annotation.
Hierarchy-guided soft targets: non-paired samples sharing organ-level findings are assigned graded positive weights rather than forced to be hard negatives, encoding anatomy-grounded semantic neighborhoods.
Query-based global branch: lightweight cross-attention aggregation of the 3D feature lattice to produce disease-relevant whole-volume representations.

Results¶

Zero-shot diagnosis (CT-RATE, 18-category macro AUROC): OKA-CT 84.9 vs. fVLM 77.8, Merlin 72.8, CT-CLIP 73.1, BIUD 71.3 — largest reported margin.
Zero-shot diagnosis (RAD-ChestCT, cross-dataset): OKA-CT 72.2 AUROC vs. COLIPRI-C 69.1, fVLM 68.0, Merlin 64.4, CT-CLIP 62.9.
CT-RATE image–image retrieval (MAP@5/10/50): OKA-CT 71.1/61.6/54.1 vs. CT-CLIP 68.3/57.2/48.9 and Merlin 62.6/51.3/43.9.
CT-RATE report–image retrieval (Recall@5/10/50/100): OKA-CT 11.9/18.6/42.9/57.1 vs. CT-CLIP 2.9/5.0/18.0/28.7 — large absolute gains.
Patch-occlusion analysis: OKA-CT representations show qualitatively stronger sensitivity to disease-associated anatomical regions compared to baselines.

Limitations¶

Organ scope is limited to four chest structures (lung, heart, esophagus, aorta); extension to full-body or non-chest CT requires additional segmentation pipelines and report parsing schema.
Organ masks depend on an automatic CT segmentation pipeline; segmentation errors propagate into organ-conditioned supervision.
Organ-hierarchical knowledge is automatically derived and not manually verified; slot quality depends on the fidelity of RadGraph-XL parsing and Qwen3-8B structuring.
Evaluated only on non-contrast chest CT; generalizability to contrast-enhanced or other modalities is untested.
The soft target weighting hyperparameters (\(\alpha, \beta, \gamma, \sigma\)) require tuning and their sensitivity is not fully analyzed in the available text.

Relevance to Vision-Language Models¶

OKA-CT advances medical VLP by demonstrating that structured semantic knowledge extracted from text (via LLM-assisted parsing) can supervise both the visual encoder and the contrastive alignment objective — a paradigm directly applicable to general VLMs operating in structured domains. The soft-target contrastive formulation, which replaces hard negative assignments with hierarchy-derived semantic neighborhoods, is a broadly relevant technique for any VLM trained on naturally clustered or hierarchically annotated data. For researchers tracking VLMs, this work exemplifies how domain-specific text structure (radiology report organization) can be converted into supervisory signal without human annotation, extending contrastive learning beyond simple paired positive/negative paradigms. It also highlights the value of disentangling visual representation learning from alignment fine-tuning — a staging strategy applicable in general-purpose VLM pretraining.