Skip to content

Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning

🕒 Published (v1): 2026-06-23 13:35 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Jolia is a 3D CT foundation model trained with ConQuer (Concept Queries), a vision-language pretraining method that replaces single-vector CLIP alignment with per-concept contrastive losses — one per anatomical region — learned via cross-attention pooling without segmentation masks. It consistently outperforms CLIP-style and segmentation-based baselines on findings classification, cross-center transfer, and report generation across chest and abdominal CT.

Problem

Standard CLIP-style pretraining compresses an entire CT volume and its multi-section radiology report into single global vectors. This loses organ-level structure critical in radiology, where reports are organized finding-by-finding across dozens of organs. Existing alternatives either use segmentation masks (limiting coverage to supported organs) or unstructured token-level alignment that lacks explicit anatomical anchoring.

Method

ConQuer augments global CLIP with \(K\) independent per-concept contrastive losses (one per anatomical region). Three components:

  1. LLM-based report decomposition: GPT decomposes each report into atomic, organ-tagged findings and groups them by a pre-defined anatomical taxonomy (\(K=102\) concepts for chest+abdomen). The same pipeline extracts 252-label finding taxonomies for evaluation.

  2. Cross-attention pooling: \(K\) learnable query tokens \(q_k\) attend over multi-scale patch tokens \(z^{(s)}\) from the image encoder via single-head cross-attention at each scale \(s\). Per-scale outputs concatenate to \(c_k \in \mathbb{R}^{576}\), adding only \(\sim\)0.6 M parameters (\(\sim\)3% of the Atlas backbone).

  3. Training objective: A weighted sum of global CLIP and per-concept InfoNCE losses: $\(L = \lambda_\text{CLIP} L^\text{[CLS]}_\text{CLIP} + \lambda_\text{ConQuer} \cdot \frac{1}{|K_b|}\sum_{k \in K_b} L^{(k)}_\text{CLIP}\)$ with \(\lambda_\text{CLIP}=1.0\), \(\lambda_\text{ConQuer}=0.5\). A concept presence mask excludes empty text sections; each query learns purely from contrastive signal with no spatial supervision.

Text encoder is frozen Qwen3-Embedding-8B (projected to \(\mathbb{R}^{576}\)). Backbone is 3D Atlas transformer (22 M params). Trained on 74,434 paired CT–reports (CT-RATE, INSPECT, Merlin-Abd-CT) on 8× H100 for 120,000 steps.

Downstream, Jolia exposes \([\mathbf{z}_\text{[CLS]}; c_k]\) (global + concept-anchored) for a specific finding's concept \(k\).

Key Contributions

  • ConQuer: concept-level contrastive pretraining without any segmentation mask or spatial annotation, generalizable across architectures and concept taxonomies.
  • Jolia: 3D CT foundation model trained on 74,434 public pairs, with publicly released weights.
  • Standardized zero-shot evaluation: prompt-aggregation protocol to remove template-selection bias; long-form prototype-based zero-shot using averaged report embeddings.
  • Built-in attention maps providing spatial interpretability per anatomical concept as a byproduct of query training.

Results

Linear probing (AUROC): - Jolia average across 4 benchmarks: 84.12 vs. CLIP Baseline 82.42, Pillar-0-Best 82.92, SPECTRE 80.69 - Gains over Baseline CLIP: +1.7 average; largest on OOD splits (+2.4 EXT-Abd-CT, +1.5 EXT-Chest-CT) - +2.2 AUROC over Pillar-0-Chest on CT-RATE; +2.0 over SPECTRE on EXT-Abd-CT - Beats segmentation-based fVLM by +8.2/+9.4 on chest (without providing masks to fVLM at test time)

Cross-center transfer (AUROC, Avg EXT): - Jolia: 75.88 (Jolia-[CLS]: 77.05) vs. Pillar-0-Best 75.00, SPECTRE 72.01

Zero-shot: - Short: Jolia leads abdomen, trails COLIPRI on chest by ~3–4 AUROC (COLIPRI trained with dedicated zero-shot augmentation) - Long (r=50 prototypes): Jolia 79.05 on CT-RATE (vs. Pillar-0-Chest 78.60); 83.03 on Merlin-Abd-CT (behind Merlin 84.26)

Ablation: - Query-only token competitive with full [CLS] baseline; combined [z[CLS]; ck] is best in every (architecture, dataset) cell - ConQuer gains hold across Atlas, ResNet-101 backbones; ViT-B trails by 3–6 AUROC and gains less

Limitations

  • Concept taxonomy must be pre-defined; coverage is fixed at 102 concepts — findings outside the taxonomy receive no per-concept pooling.
  • ViT-B backbone does not reliably benefit from ConQuer, suggesting multi-scale features are important for the cross-attention queries.
  • Cross-center performance on EXT-Chest-CT trails Pillar-0, attributed to pretraining data quality differences rather than the method itself.
  • Report decomposition relies on a proprietary LLM (GPT); reproducibility of the preprocessing pipeline depends on API access.
  • Zero-shot evaluation reveals a bias toward "{∅/no} findings" phrasing from pretraining; short-prompt zero-shot remains sensitive to template wording.
  • Per-concept tokens are not used in report generation; the generative head relies only on \(z_\text{[CLS]}\).

Relevance to Foundation Models in Medicine

Jolia directly advances the 3D medical foundation model literature by demonstrating that concept-level contrastive alignment — without any pixel-level supervision — consistently improves over global CLIP on in- and out-of-distribution clinical benchmarks, a critical requirement for real-world deployment. The ConQuer framework offers a principled alternative to segmentation-dependent anatomical grounding (fVLM, TotalFM), removing a major data bottleneck for scaling. The built-in spatial interpretability via query attention maps and the standardized multi-prompt zero-shot protocol address two persistent pain points in evaluating medical VLMs. This work is directly relevant to researchers tracking how foundation models can be made more structure-aware for radiology without requiring additional annotation effort.