Skip to content

RELICT: A Replica Detection Framework for Medical Image Generation

🕒 Published (v1): 2025-02-24 17:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RELICT is an open-source framework for detecting replicas—near-identical copies of training images—in synthetic medical image datasets, combining voxel-level, foundation-model feature-level, and segmentation-level similarity analyses. Validated on 3D head CT (ICH) and TOF-MRA (Circle of Willis) generation tasks, it provides a standardized quality-assurance step that is currently absent from most generative medical imaging pipelines.

Problem

Generative models trained on small, sensitive medical imaging datasets can memorize and reproduce near-identical copies of patient data, violating patient privacy and reducing the diversity benefit of synthetic augmentation. No standardized, task-agnostic tool exists for detecting such replicas; prior works use ad hoc, incompatible methods (Dice overlap, custom siamese nets, pixel correlation) that cannot be systematically compared or reused.

Method

RELICT computes pairwise similarity between each synthetic image and all training images using three complementary analysis levels:

  1. Image-level: MAE, RMSE, SSIM directly on voxel intensities.
  2. Feature-level: ResNet-50 MedicalNet (pretrained on 23 segmentation datasets) encodes images into flattened (2048Ă—4Ă—4Ă—4) embeddings; similarity computed via embedding RMSE and cosine similarity.
  3. Segmentation-level: nnUNet segmentations of pathology/anatomy ROIs compared with Dice coefficient and average surface distance (ASD), in binary and multiclass settings.

For each measure, a distance ratio is computed—dividing the similarity to the closest training image by the mean similarity to the 50 nearest training images—to quantify how "abnormally close" a synthetic image is. A threshold on this ratio yields a binary replica classification. Thresholds are calibrated against expert visual scoring (4-point Likert scale, two senior raters) on 50 generated images per use case.

Key Contributions

  • Unified, open-source replica detection framework covering image-, feature-, and segmentation-level analysis in a single pipeline.
  • Demonstrates that medical foundation model embeddings (MedicalNet) can serve as fast, modality-agnostic feature extractors for replica detection.
  • Shows that no single analysis level is universally optimal: voxel/feature methods suffice for near-identical intensity replicas (NCCT), while segmentation-level analysis is superior when replicas manifest via structural/anatomical similarity (TOF-MRA).
  • Provides a ranked output of candidate replicas to guide targeted expert review and threshold tuning.
  • Proposes inclusion of replica detection in AI-in-medicine reporting checklists (CLAIM, FUTURE-AI).

Results

  • NCCT use case (latent diffusion, N=774 training images): 45/50 generated images rated as replicas by experts. Image-level (MAE, RMSE) and feature-level (embedding RMSE, cosine similarity) methods all achieved balanced accuracy = 1.0 at their optimal thresholds. Segmentation-level Dice and ASD reached 0.96 and 0.98, respectively.
  • TOF-MRA use case (3D StyleGANv2, N=1,782 training images): Only 5/50 generated images were replicas. Segmentation-level multiclass ASD achieved the best balanced accuracy of 0.79; all image-level and feature-level measures stayed below 0.72.
  • Inter-rater agreement: 92% (NCCT), 82% (TOF-MRA).
  • Runtime on typical workstation: image-level 2–76 min, feature-level 2–4 min, segmentation-level 3–31 min depending on modality and measure.

Limitations

  • Only two use cases tested, each with a single generative architecture; memorization rates cannot be compared across architectures.
  • Only one medical foundation model (ResNet-50 MedicalNet) evaluated for feature extraction.
  • Visual scoring ground truth was based on RMSE-preselected nearest neighbors only, not exhaustive pairwise comparison, potentially missing non-RMSE-nearest replicas.
  • Framework does not account for data augmentation applied during generative model training, which can shift the nearest-neighbor identity.
  • Perfect replica detection in the TOF-MRA setting was not achievable at any threshold with any tested measure.

Relevance to Foundation Models in Medicine

The paper directly operationalizes a pretrained medical foundation model (MedicalNet/ResNet-50 trained on 23 segmentation datasets) as a general-purpose feature extractor for privacy auditing of synthetic data—a use case orthogonal to the typical downstream fine-tuning paradigm. This demonstrates that foundation model embeddings carry clinically meaningful image similarity structure transferable to unseen modalities and tasks. For the broader field, RELICT raises a critical but underappreciated governance question: as foundation models are increasingly used to generate synthetic training data at scale, systematic memorization auditing using complementary foundation model probes becomes a prerequisite for responsible data sharing. The paper also motivates incorporating newer foundation models (BiomedParse, MedSAM) as drop-in feature extractors to improve detection robustness.