Federated Client-tailored Adapter for Medical Image Segmentation¶

🕒 Published (v1): 2025-04-25 02:20 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

FCA (Federated Client-tailored Adapter) addresses training instability and client heterogeneity in federated medical image segmentation by inserting lightweight adapters into frozen medical foundation models (SAM-Med2D) and dynamically decomposing adapter parameters into client-invariant (globally updated) and client-specific (locally updated) components. Two updating strategies—binary (BFU) and smooth (SFU)—enable per-client optimal models without sharing raw data. FCA-SFU achieves state-of-the-art mDice on three heterogeneous X-ray/OCT datasets.

Problem¶

Centralized medical image segmentation cannot operate under privacy constraints that prohibit raw data sharing across hospitals. Conventional federated learning (FL) applied to medical segmentation suffers from: (1) heavy training instability and slow convergence due to client-wise heterogeneity (class imbalance + distribution diversity across data islands); and (2) convergence to a globally compromised model that is suboptimal for any individual client. Existing personalized FL methods decouple at coarse-grained levels and assign parameter groups statically, lacking fine-grained adaptability.

Method¶

FCA inserts adapter layers (borrowed from SAM-Med2D's PEFT pipeline) into each transformer layer of a frozen medical foundation model encoder. Most MFM weights are frozen; only adapters, prompt encoder, and mask decoder are trained—limiting the federated communication surface to a small parameter set.

Global-local Decomposer (GLD): Each adapter layer's units (channels for Conv, neurons for FC) are scored by a lightweight client discriminator trained via a domain-classification pretext task with a gradient reversal layer (GRL) to prevent leakage into the main branch. The contribution score S ∈ ℝ^{U×K×K} quantifies how much each unit's activation is discriminative for each client.

Binary Federated Updating (BFU): Normalized entropy of each unit's score distribution (D_{u,k}) determines global vs. local assignment via threshold δ=0.25. Global units are averaged across clients; local units are updated only locally.

Smooth Federated Updating (SFU): Instead of binary assignment, each unit's update is a weighted average over all clients' parameters, weighted by the unit's contribution score toward each client—providing a soft, probabilistic generalization of BFU.

Key Contributions¶

Identification of training instability in FL for medical segmentation caused by client-wise class imbalance and distribution diversity, with MFM adapters as a stabilization mechanism.
FCA framework: PEFT adapters tightly coupled with FL, distilling universal MFM knowledge while enabling privacy-preserving distributed training.
Global-local Decomposer (GLD) with a client-discriminator pretext task for dynamic, fine-grained unit-level decomposition of adapter parameters.
Two client-tailored updating strategies (BFU and SFU), where SFU provides probabilistic soft weighting as a generalization of binary hard assignment.
SOTA mDice on three heterogeneous distributed datasets (CXRS-HG, HLS, AMD-SD-HG) across diverse anatomies and imaging modalities.

Results¶

CXRS-HG (chest X-ray, 30-class anatomy segmentation): FCA-SFU 64.15 avg mDice vs. best baseline IOP-FL* 62.00 (+3.51 pp); FCA-BFU 63.41.
HLS (heterogeneous lung segmentation, 4 real COVID-19 datasets): FCA-SFU 92.44 avg mDice vs. IOP-FL* 91.68 (+1.65 pp best per-client up to 93.98 on Client3.
AMD-SD-HG (OCT retinal segmentation): FCA-SFU 63.40 avg mDice vs. IOP-FL* 61.41 (+1.99 pp).
FCA-SFU consistently outperforms FCA-BFU across all datasets/clients.
MFM-enhanced FedAvg* (60.64) substantially outperforms vanilla FedAvg (36.94) on CXRS-HG, confirming the stabilizing effect of frozen MFM priors alone.
Generalization confirmed across three MFMs: H-SAM (62.15), Med-SA (51.53), SAM-Med2D (64.15 avg); FCA improves over the FedAvg* counterpart in each case.
FCA-SFU improves over FedAvg* under IID (+1.19), class imbalance (+2.49), distribution diversity (+1.14), and combined heterogeneity (+3.51) conditions.

Limitations¶

Evaluated exclusively on X-ray images (and one OCT dataset); generalization to CT, MRI, pathology remains unvalidated.
Communication overhead from transmitting contribution scores S to the central server each round is not quantified or compared against baselines.
Threshold δ=0.25 for BFU is empirically set; sensitivity analysis is not discussed in the available text.
Experiments use simulated heterogeneity (Dirichlet splits + image transformations) for CXRS-HG and AMD-SD-HG; only HLS uses genuinely multi-site data.
The client discriminator introduces an additional hyperparameter (GRL scale) and training objective not analyzed under varying numbers of clients or extreme imbalance.

Relevance to Foundation Models in Medicine¶

This paper demonstrates a concrete strategy for deploying medical foundation models (SAM-Med2D and variants) under federated, privacy-constrained settings—a critical real-world deployment barrier for MFMs. By freezing the bulk of MFM parameters and federating only lightweight adapters, the work shows that MFM priors can actively stabilize FL convergence under severe data heterogeneity, not merely boost final accuracy. The GLD mechanism is a novel adapter-level decomposition technique that may generalize to other PEFT-based MFM adaptation scenarios beyond FL. For researchers tracking foundation models in medicine, FCA charts a path toward clinically deployable MFMs that respect data governance constraints across hospital networks.