Skip to content

Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation

🕒 Published (v1): 2025-04-03 07:38 UTC · Source: Arxiv · Venue: CVPR · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a multi-teacher knowledge distillation framework that agglomerates complementary medical vision foundation models (MedSAM2, RAD-DINO, MedCLIP) into a single lightweight student encoder for videofluoroscopy swallowing study (VFSS) segmentation. The approach closes the performance gap between heavy foundation models and efficient lightweight alternatives without adding inference-time parameters. The distilled student encoder is up to 187Ă— smaller than MedSAM2 while achieving a 2% Dice improvement over single-teacher distillation.

Problem

Large medical foundation models (MedSAM2, RAD-DINO, MedCLIP) achieve strong segmentation performance but are too large for real-time inference. Existing lightweight distilled variants (EfficientViT, TinyViT, RepViT) suffer a substantial performance gap versus the teacher models because single-teacher distillation cannot capture the full breadth of visual features—modality understanding, spatial localization, and segmentation—encoded across different foundation models.

Method

The framework distills three complementary teacher encoders simultaneously into one lightweight student encoder (TinyViT/RepViT/EfficientViT backbone + SAM2 decoder):

  • Teachers: MedCLIP (modality understanding), RAD-DINO (localization), MedSAM2 (segmentation)
  • Projection heads: Per-teacher MLP heads map student patch tokens into each teacher's embedding space; distillation loss is cosine similarity or MSE between projected student features and teacher patch tokens (not CLS tokens)
  • Loss balancing: Two learned strategies to weight per-teacher losses adaptively:
  • MLP-based: a small MLP over the loss vector produces softmax weights
  • Attention-based: scaled dot-product attention between a learnable query (from student embedding) and per-teacher key vectors yields weights
  • Feature standardization: PHI-S normalization applied to teacher embeddings to equalize distribution variance before distillation, preventing any single teacher from dominating
  • Training is two-stage: 100 epochs distilling the encoder, then 100 epochs fine-tuning the decoder on VFSS-5K segmentation ground truth

Key Contributions

  • First application of multi-teacher agglomeration distillation to medical image segmentation
  • Attention-based loss balancing mechanism that adaptively weights teacher contributions using student-query/teacher-key dot-product attention, requiring only minimal additional parameters
  • Combination of PHI-S feature standardization with adaptive loss balancing for stable multi-teacher training
  • Empirical demonstration across 12 anatomical structures in VFSS-5K: efficient models (5–31M params) achieve competitive performance versus a 224M-parameter specialist

Results

  • Best distilled student (TinyViT-m): avg. Dice 0.826, avg. HD95 12.641 on VFSS-5K (12 structures)
  • Compared to baseline single-teacher TinyViT distillation: Dice 0.804 → 0.826 (+2.2%), HD95 14.436 → 12.641
  • RepViT-m: Dice 0.762 → 0.781; EfficientViT-m: Dice 0.783 → 0.801
  • MedSAM2 specialist (224.4M): Dice 0.849, HD95 8.974—still outperforms efficient models but requires per-task fine-tuning
  • Attention-based loss balancing alone (Row 4 vs Row 1 ablation): Dice 0.818 vs 0.800 (+1.8%)
  • PHI-S standardization + attention balancing (Row 5): Dice 0.826, HD95 12.641 (best configuration)
  • Adding MedCLIP as third teacher yields only marginal gain over MedSAM2+RAD-DINO (0.826 vs 0.823), attributed to low inter-cluster cosine distance (0.21) in VFSS domain

Limitations

  • Evaluated on a single medical imaging modality (videofluoroscopy/X-ray video); generalization to other modalities (MRI, CT, histology) not demonstrated
  • MedCLIP contributes marginally in this narrow-domain dataset—its utility may not transfer to other settings without larger, more diverse data
  • Student models still lag behind the fine-tuned MedSAM2 specialist (Dice ~0.83 vs ~0.85), particularly on surface distance (HD95 12.6 vs 8.97)
  • Sequential application of teacher embeddings and task-specific encoder training are left as future work
  • Evaluation dataset (VFSS-5K) is single-institution; no external validation

Relevance to Foundation Models in Medicine

This work directly addresses the deployment bottleneck of large medical foundation models by showing that their complementary specializations can be systematically harvested into a single efficient encoder via multi-teacher distillation—a practical pathway for resource-constrained clinical deployment. It demonstrates that no single medical foundation model captures all necessary visual priors for segmentation, motivating ensemble/agglomeration strategies over single-model fine-tuning. The attention-based teacher weighting is a transferable technique applicable to any multi-foundation-model distillation setup in medical imaging. The finding that MedCLIP contributes minimally in narrow-domain settings offers useful guidance on teacher selection for task-specific medical foundation model compression.