Skip to content

Towards All-in-One Medical Image Re-Identification

🕒 Published (v1): 2025-03-11 08:35 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MaMI is a unified medical image re-identification (ReID) model that handles diverse imaging modalities (X-ray, CT, MRI, fundus, histopathology) with a single network. It uses a runtime parameter adapter to generate modality-specific LoRA weights on the fly, and distills inter-image differential features from pre-trained medical foundation models (MFMs). Evaluated against 25 foundation models and 8 large VLMs across 11 datasets, MaMI achieves state-of-the-art performance on all modalities.

Problem

Existing MedReID methods are single-modality (e.g., only chest X-ray or only fundus), cannot exploit cross-modality training data, and lack clinically meaningful priors — making them brittle and hard to generalize. Naive multi-modality fine-tuning with a shared model hurts performance on some modalities (e.g., X-ray drops from 94.21% to 92.30%) because modality-agnostic features dominate. No unified benchmark existed for comparing approaches across modalities.

Method

Continuous Modality-based Parameter Adapter (ComPA): Given an input image, patch features are processed by an MLP to produce a global modality context, then mapped to a probability vector over L=32 learnable modality basis vectors (a soft codebook). A weighted sum produces a continuous modality embedding M_i, which two networks (FFN-PNet, Att-PNet) map to low-rank LoRA parameters. These parameters are merged layer-wise into a shared ViT-Base backbone at inference time, effectively turning one modality-agnostic model into a modality-specific one per input — without discrete modality labels.

Medical Prior Transfer via Inter-Image Differential Alignment: Modality-specific query tokens (derived from M_i via MLP) attend to the local feature map to select N key feature vectors (e.g., rib shape, vessel patterns) from both the MaMI backbone and the matching pre-trained MFM (Med-UniC for X-ray, CT-CLIP for CT, RetFound for fundus, CHIEF for histopathology). Instead of aligning single-image features, the method aligns pairwise differences P_i − P_j (from MaMI) to Q_i − Q_j (from MFM) using a contrastive loss (temperature τ=0.07). This mirrors the ReID objective of discriminating image pairs.

Training losses: cross-entropy ID classification + triplet loss (identity terms) + λ·L_med-align (medical prior term), λ=0.01.

Key Contributions

  • First all-in-one MedReID model (MaMI) handling X-ray, CT, MRI, fundus, and histopathology with a single ViT-Base backbone.
  • ComPA: instance-adaptive continuous modality representation that generates LoRA parameters at runtime, outperforming both discrete-label conditioning and naive multi-modality fine-tuning.
  • Inter-image differential feature alignment with a bag of MFMs, transferring anatomical priors to the ReID task without adding inference cost.
  • Comprehensive benchmark: 11 datasets, 25 visual foundation models, 8 large VLMs, and single-modality MedReID baselines compared under a unified protocol.
  • Two real-world deployment demos: longitudinal history-augmented diagnosis and visual identity removal for privacy protection.

Results

  • MaMI CMC-R1 on all 11 datasets (Table 2): 96.89% (MIMIC-X), 95.01% (Chest-X), 88.09% (CCII lung CT), 82.68% (HCC-TACE AB-CT), 76.82% (KIRC), 85.00% (OASIS2 brain MRI), 85.71% (Mess2 fundus), 71.34% (ODIR fundus), 71.00% (GRAPE fundus), 68.75% (LUAD histopathology) — best across all modalities.
  • vs. best fine-tuned VL baseline CLIP†: +3.87% on MIMIC-X (96.89 vs. 93.52), +22.00% on Chest-X (95.01 vs. 73.00), +17.00% on OASIS2 (85.00 vs. 68.00).
  • vs. best single-modality MedReID Packhäuser et al.: +4.47% on MIMIC-X; 95.01% vs. 53.62% on Chest-X (generalizes where specialist fails).
  • vs. large VLMs on Chest-X: MaMI 98.80% vs. QWen-VL-Max 76.80%, GPT-4o 62.50%.
  • Cross-modality ReID (X-ray↔CT on 1814-patient respiratory cohort): MaMI all-in-one 87.28%, fine-tuned 94.38%, vs. single-modality MaMI 76–78%.
  • History-augmented diagnosis: retrieving 5 historical images improves accuracy from 77.34% to 80.12% (+2.78%).
  • Privacy protection: ReID accuracy on protected Chest-X drops from 91.49% to 21.23% for MaMI; protected images also resist other models (CLIP†: 11.86%, Med-Unic†: 13.94%); disease classification accuracy barely drops (81.24% → 80.67%).
  • Ablation (Table 5): baseline CLIP→96.89% requires both ComPA (+4.31% X-ray, +11.91% Ab-CT) and MFM alignment (additional +2.14% on Chest-X, +1.55% on GRAPE, notably on small datasets like HCC-TACE: 80.95%→88.09%).

Limitations

  • MFM selection is modality-contingent: a separate, modality-matched MFM must exist (Med-UniC, CT-CLIP, RetFound, CHIEF); new modalities require an appropriate MFM.
  • Multi-slice handling is simple mean-pooling over 8 uniformly sampled slices; more sophisticated 3D aggregation is acknowledged but not implemented.
  • The privacy removal U-Net is trained only on MIMIC-X (X-ray) and evaluated on Chest-X; broader multi-modality privacy protection is not demonstrated.
  • Cross-modality evaluation uses one private licensed dataset; public cross-modality benchmarks are lacking.
  • Benchmark coverage of histopathology is limited (LUAD only, 542 training images), and performance there (68.75%) lags other modalities.

Relevance to Foundation Models in Medicine

MaMI directly operationalizes a "bag of MFMs" paradigm — rather than relying on a single universal model, it selectively extracts anatomical priors from modality-specialized MFMs (CXR, CT, fundus, pathology) and distills them into a downstream task via differential contrastive alignment. This is a concrete blueprint for how community-built domain MFMs can be composed rather than fine-tuned end-to-end, which is highly relevant to researchers studying MFM aggregation, knowledge transfer, and generalization. The ComPA mechanism offers a practical solution to modality heterogeneity that generalizes beyond ReID to any task requiring a single model across radically different imaging domains. The real-world deployment experiments (privacy and longitudinal care) also demonstrate how MFMs can anchor clinical utility in emerging non-diagnostic application categories.