Skip to content

Learning Directional Semantic Transitions for Longitudinal Chest X-ray Analysis

🕒 Published (v1): 2026-06-14 17:42 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ProTrans is a vision-language pretraining framework that models chest X-ray disease progression as a directional semantic transition between paired studies, rather than as feature fusion or subtraction. It introduces a learnable progression feature map between prior and current CXR embeddings, aligned to radiology report-derived state and transition descriptions, with reversed temporal modeling to enforce trajectory directionality. It outperforms all baselines on both progression classification and captioning tasks.

Problem

Existing longitudinal CXR methods either fuse temporal features (cross-attention, alignment connectors) or model inter-study discrepancies (patch-level difference) but fail to capture (1) subtle directional progression semantics overwhelmed by invariant anatomy and pseudo-changes from imaging variability, and (2) the inherently directional nature of disease trajectories (improving vs. worsening), which prior work largely ignores.

Method

ProTrans inserts a learnable progression feature map \(P_I^\rightarrow \in \mathbb{R}^{N_I \times d}\) between prior and current CXR embeddings to form a directed temporal sequence \(S^\rightarrow = [H_I^\text{pri}; P_I^\rightarrow; H_I^\text{cur}]\). This sequence is processed by a progression encoder consisting of \(L=3\) spatiotemporal blocks, each with interleaved temporal attention (cross-time dependencies), spatial attention (anatomical correspondence), and an FFN.

Three training objectives jointly supervise the representation:

  1. State contrastive alignment (\(\mathcal{L}^\text{MoD}_\text{state}\)): Cross-modal InfoNCE between each CXR embedding and its paired report, treating same-sample cross-timepoint reports as negatives to enforce fine-grained temporal discrimination. Soft targets from a momentum EMA model mix with hard one-hot labels (weight \(\alpha=0.8\)).

  2. Transition contrastive alignment (\(\mathcal{L}^\text{MoD}_\text{trans}\)): Aligns the progression feature \(\tilde{P}_I^\rightarrow\) to structured text descriptions in the form "[finding] at [position] [progression]" from Chest ImaGenome annotations; reversed-direction descriptions (e.g., "improved"↔"worsened") serve as hard negatives, enforcing directionality.

  3. Bidirectional reconstruction (\(\mathcal{L}_\text{recon}\)): A reversed progression map \(P_I^\leftarrow\) is also learned. A lightweight decoder reconstructs each state from the opposite state plus its progression representation; token-level InfoNCE loss enforces consistency. This disentangles directional semantics and ensures forward/backward coherence.

Overall objective: \(\mathcal{L} = \mathcal{L}^\text{MoD}_\text{state} + \mathcal{L}^\text{MoD}_\text{trans} + \lambda_r \mathcal{L}_\text{recon}\) (\(\lambda_r=0.5\)). Backbone: ViT-B/16 + BioClinicalBERT; pretrained on 98,940 bi-temporal MIMIC-CXR-JPG pairs with Chest ImaGenome progression annotations.

Key Contributions

  • Formulation of longitudinal CXR analysis as directional semantic transition learning via a learnable progression feature map inserted into the temporal sequence.
  • Joint state and transition contrastive alignment grounding visual states and progression features in a shared semantic space using radiology reports and structured progression annotations.
  • Reversed temporal modeling and bidirectional reconstruction consistency that explicitly enforce trajectory directionality and disentangle forward/backward semantics.
  • Unified pretraining framework demonstrated effective on both disease progression classification (MS-CXR-T) and progression captioning (ICG dataset).

Results

Disease Progression Classification (MS-CXR-T, 5 findings, 3-class): - SVM setting: ProTrans avg. 63.54% vs. BioViL-T 61.12%, MedST 53.93%, Diff-RRG 63.54% (tied on Pneumothorax; ProTrans best overall avg.) - Text prototype setting: ProTrans avg. 65.81% vs. BioViL-T 56.30%, MedST 62.66%, TempA-VLP 64.8%, Coca-CXR 65.0%, Diff-RRG 53.36%

Progression Captioning (ICG dataset, N=760 test): - ROUGE-L: 0.259 vs. best baseline Libra 0.190 (+36%) - BERTScore: 0.535 vs. Libra 0.428 (+25%) - GREEN: 0.230 vs. Libra 0.141 (+63%) - Temporal-F1: 0.238 vs. Libra 0.145 (+64%) - RadGraph-F1: 0.126 vs. Libra 0.081 (+56%) - CheXbert: 0.386 vs. LLaVA-Rad 0.383 (marginal)

Ablation (SVM avg. accuracy): - Removing \(\mathcal{L}^\text{MoD}_\text{trans}\): largest drop, 53.65% (−10pp) - Removing \(\mathcal{L}^\text{MoD}_\text{state}\): 60.64% - Removing \(\mathcal{L}_\text{recon}\): 61.70% - Removing bidirectional: 61.98% - Full model: 63.54%

Limitations

  • Evaluated only on CXR; generalization to other longitudinal medical imaging modalities (CT, MRI) is not demonstrated.
  • Progression captioning evaluation uses BioMistral-7B as a frozen LLM; caption quality is partially bounded by that LLM's medical language capabilities.
  • The reversed transition text is constructed by simple antonym substitution ("improved"↔"worsened"), which may not cover nuanced clinical trajectory descriptions.
  • Pretraining requires disease progression annotations from Chest ImaGenome beyond standard image-report pairs, limiting scalability to datasets without such structured annotations.
  • No patient-level long-term (>2 timepoints) longitudinal modeling; only bi-temporal pairs are handled.

Relevance to Vision-Language Models

ProTrans is a direct extension of VLP contrastive pretraining (à la CLIP/BioViL) to the temporal domain, showing how report-grounded contrastive objectives can be extended beyond single-image alignment to encode directional semantic transitions. The momentum distillation strategy (soft targets from EMA model) and structured text prototype design for zero-shot-style classification are directly applicable techniques in medical VLM design. The work highlights that standard VLM pretraining with single-study image-text pairs is insufficient for temporal reasoning tasks, motivating richer temporal alignment objectives in the broader VLM research agenda. The progression captioning results—where a frozen general medical LLM gains strong temporal awareness purely from learned visual progression features—demonstrate the value of temporally-grounded visual representations as plug-in encoders for VLM pipelines.