Skip to content

SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI

🕒 Published (v1): 2025-03-25 16:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SeLIP is a contrastive language-image pretraining framework for multi-modal head MRI that replaces the standard hard identity-matrix target with a mixed syntax-and-semantic soft similarity target, enabling effective pretraining on small Chinese-language clinical datasets. A locally finetuned LLM (Qwen2-7B) first extracts structured per-modality findings from free-text radiology reports, which are then used as paired training signals. The framework outperforms vanilla CLIP across image-text retrieval, stroke classification, and brain tumor segmentation.

Problem

Standard CLIP-style contrastive learning treats all non-paired image-text samples as hard negatives (target = identity matrix). In radiology, many reports share identical or near-identical findings across patients, making this hard-negative assumption incorrect and degrading representation quality—a problem exacerbated by the small scale of available medical image-report datasets and the multi-modal nature of MRI (where a single report describes findings across T1WI, T2WI, DWI, ADC, and T2FLAIR jointly, not per-slice).

Method

  1. Structured report extraction: A Qwen2-7B model finetuned via LoRA on pseudo-data (GPT-4-turbo-generated reports from random JSON items) converts free-text Chinese radiology findings into per-modality JSON tuples {modality, orientation, anatomic site, appearance}, yielding clean image-specific text descriptions.
  2. CLIP-like pretraining backbone: ResNet (18 or 50) image encoder + BERT (Chinese) text encoder; embeddings projected to 512-dim; standard InfoNCE loss L_clip.
  3. Mixed syntax-and-semantic similarity target: For each pair of text descriptions in a batch, a soft similarity score is computed as s̃_{i,j} = ½ · TDC(t_i, t_j) · (w_loc(t_i,t_j) + w_per(t_i,t_j)), where TDC is text Dice coefficient (syntax overlap), w_loc is anatomic-site match, and w_per is appearance match. This yields a batch-level soft similarity matrix S.
  4. KL divergence loss: L_se minimizes KL divergence between the image-text cosine similarity distribution and S (column-normalized), for both image-to-text and text-to-image directions.
  5. Total loss: L = α·L_clip + β·L_se. Dataset: 10,825 subjects from Tianjin First Central Hospital (TFCH); 17,529 training image-text pairs; reports in Chinese.

Key Contributions

  • Contrastive pretraining framework specifically designed for 3D multi-modal head MRI with Chinese-language reports.
  • Mixed syntax-semantic soft target that avoids penalizing genuinely similar but unpaired report descriptions.
  • Privacy-preserving local LLM pipeline (Qwen2-7B + LoRA) for structured report extraction, achieving 94.59% accuracy vs. 52.38% for GPT-4-turbo on this task.
  • Demonstrated transfer to a never-seen MRI modality (T1ce on BraTS19) without modality-specific pretraining.

Results

Image-text retrieval (TFCH test set, 1,689 unique texts): - ResNet50 SeLIP vs. CLIP: Top-1 0.2871 vs. 0.2283; Top-10 0.3514 vs. 0.3303 - ResNet18 SeLIP vs. CLIP: Top-1 0.2809 vs. 0.2059; Top-10 0.3423 vs. 0.3007

Stroke classification (same-domain DWI, 400-sample test set): - SeLIP outperforms CLIP in nearly all zero-shot, finetuning, and low-data (10%/25% training split) conditions for both ResNet18 and ResNet50; pretrained models consistently exceed train-from-scratch baselines.

Cross-domain glioma grading (BraTS19, unseen T1ce modality): - SeLIP-ZS achieves +35.65% ACC, +27.78% F1, +32.98% AUC over KAD baseline. - SeLIP-FT improves F1 by 3.23% over DSM (train-from-scratch) model.

Structural information extraction: - Qwen2-7B finetuned: 99.50% parsing success rate, 94.59% accuracy vs. GPT-4-turbo (77.00%, 52.38%).

Limitations

  • Dataset is single-institution (TFCH), Chinese-language only; generalizability to other languages, imaging protocols, and hospital systems is unverified.
  • Image-text pairs are substantially fewer than raw scans (e.g., T2FLAIR: 5,952 scans → 1,142 pairs) due to pairing constraints, reducing effective training data.
  • Text encoder (BERT-base-Chinese) and image encoder (ResNet) are relatively small; no exploration of larger vision-language backbones (e.g., ViT-based or large VLMs).
  • Soft similarity relies on structured extraction accuracy; errors in LLM extraction propagate into incorrect soft targets.
  • Cross-modal generalization tested only on one external dataset (BraTS19); no evaluation on other MRI body regions or pathologies beyond stroke and glioma.
  • Batch size of 64 is small for contrastive learning; impact of scaling not studied.

Relevance to Foundation Models in Medicine

SeLIP directly addresses the data-scarcity and report-diversity challenges that make adapting CLIP-style foundation models to clinical MRI difficult—problems distinct from the large-scale chest X-ray literature (MIMIC-CXR, CheXpert) that dominates medical VLM research. The soft-target KL formulation is a practical contribution applicable to any domain where report redundancy violates the hard-negative assumption, making it relevant to CT, ultrasound, and other modalities with stereotyped reporting. The local LLM-based structured extraction pipeline offers a privacy-compliant alternative to commercial APIs for report preprocessing, a recurring constraint in medical foundation model development. For researchers tracking this topic, SeLIP represents a blueprint for building institution-scale VLMs from clinical MRI archives without requiring large annotated datasets or internet-scale pretraining.