FMIR, a foundation model-based Image Registration Framework for Robust Image Registration¶

🕒 Published (v1): 2026-01-24 17:05 UTC · Source: Arxiv · Venue: ISBI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

FMIR adapts frozen 2D foundation model encoders (DINO, SAM) for 3D medical image registration by extracting domain-invariant features and pairing them with a multi-scale pyramid registration head. A channel regularization strategy—random channel dropout during training, PCA during inference—prevents the model from overfitting to dataset-specific feature patterns. Trained on a single dataset, FMIR achieves SOTA in-domain performance while generalizing to unseen imaging domains and modalities.

Problem¶

Deep learning-based registration models assume i.i.d. train/test distributions, causing performance collapse on out-of-domain data—a critical problem given the small scale of most medical datasets. Existing foundation model approaches for registration (e.g., uniGradICON) require complex architectures and slow inference. No prior work had efficiently adapted general-purpose 2D vision foundation models (SAM, DINO) for generalizable 3D medical image registration under resource constraints.

Method¶

FMIR has two components:

Foundation Model-based Encoder: Each 3D volume is sliced along the depth axis; each 2D slice is independently processed by a frozen foundation model encoder (DINO ViT-B by default). Features (c=768 for DINO, 256 for SAM) are reduced to c′=256 via channel regularization, then reconstructed to 3D via reassembly/upsampling, followed by a 3-layer 3D conv block to recover volumetric context and compress to n=32 channels.

Channel Regularization (CR): During training, a random subset of the c′=256 channels is selected per forward pass (channel dropout), preventing reliance on any fixed feature subset and suppressing dataset-specific priors. During inference, PCA deterministically projects to c′ dimensions preserving maximum variance. The train/test feature discrepancy forces learning of essential structural correspondences rather than superficial statistics.

Registration Head: A five-level coarse-to-fine feature pyramid; at each level, a 3-layer conv block predicts a residual deformation field ui, composed progressively from coarse to fine. Loss combines NCC + diffusion-based smoothness regularization, with optional Dice loss for weakly supervised training.

Key Contributions¶

Foundation model-based registration framework (FMIR) that generalizes across imaging domains when trained on a single dataset
Slice-based 3D adaptation pipeline for frozen 2D foundation encoders (DINO/SAM), with plug-and-play backbone swapping requiring no retraining
Channel regularization strategy (random selection during training, PCA during inference) that explicitly decouples train/test feature distributions to suppress dataset-specific overfitting
Multi-scale pyramid registration head compatible with any foundation encoder output

Results¶

In-domain (ACDC cardiac MR, intra-subject): - FMIR (DINO): Dice 79.82% vs. next-best RDP 78.06%, uniGradICON 78.89% - HD95: 9.07 mm; SDlogJ: 0.049 - Inference time: 0.62 s vs. uniGradICON 4.95 s (8× faster) - FMIR(SAM), zero-shot backbone swap: Dice 79.72%, HD95 9.06 mm

Cross-domain generalization (Table 2, unsupervised): - FMIR trained on Abdomen CT only → ACDC cardiac: Dice 69.18% (vs. TransMorph trained directly on ACDC: Dice ~74.97%), showing meaningful cross-domain transfer - FMIR trained on ACDC only → Abdomen CT: Dice 37.12% (DINO) / 40.20% (SAM) - Removing CR ("-CR") degrades out-of-domain Dice substantially, especially with SAM backbone

Ablation: - Weakly supervised training improves in-domain but reduces out-of-domain generalization relative to unsupervised training - Hybrid (ACDC+Abdomen) training underperforms single-dataset training on each respective domain

Limitations¶

Only evaluated on 2D-to-3D adaptation via slice-wise processing; true 3D spatial context within slices is lost during foundation model encoding and only partially recovered by the downstream 3D conv block
Cross-domain performance (e.g., abdomen→cardiac) still lags significantly behind in-domain training (Dice 69% vs. 79%)
Weakly supervised training hurts generalization, limiting practical use of anatomical label supervision in multi-domain settings
Hybrid multi-domain training degrades performance, suggesting the framework does not yet handle heterogeneous motion patterns well
Training used only DINO ViT-B; more powerful or medical-specific encoders not evaluated

Relevance to Foundation Models in Medicine¶

FMIR demonstrates a resource-efficient paradigm for repurposing general-purpose 2D vision foundation models (DINO, SAM) for a 3D medical analysis task without medical pretraining or full fine-tuning—relevant to the broader question of how much transfer value off-the-shelf foundation models carry into clinical settings. The channel regularization technique is a lightweight, architecture-agnostic mechanism for suppressing domain-specific feature overfitting, applicable beyond registration to other foundation model adaptation scenarios in medicine. The plug-and-play backbone design (SAM and DINO produce comparable results without retraining) is a concrete step toward modular, generalizable medical imaging pipelines. This sits at the intersection of parameter-efficient adaptation and domain generalization, two active themes in medical foundation model research.