Skip to content

FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FedMGP is a personalized federated prompt learning framework for VLMs that replaces the standard single text-only prompt with multiple groups of paired text-visual prompts per client. A diversity loss enforces specialization across groups, and a softmax-weighted similarity-guided sampling strategy selectively aggregates the most globally aligned prompt groups each round. It achieves state-of-the-art performance on federated VLM benchmarks while communicating only 5.1k parameters—the lowest among all compared methods.

Problem

Existing federated prompt learning (FPL) methods use a single, text-only prompt per client, which (1) cannot capture diverse visual cues or multiple semantic concepts coexisting in a client's local data, and (2) introduces biased global aggregation because a single prompt overfit to dominant local patterns cannot faithfully represent the client's full distribution. The result is a fundamental tension: methods that maximize local personalization collapse on generalization, and vice versa.

Method

Each client maintains G groups of paired text-visual prompts {p_{t,j}, p_{v,j}}. During local training: - Multimodal co-learning: for each group j, text prompt p_{t,j} is concatenated with class embeddings and passed through the CLIP text encoder; the image is prepended with visual prompt p_{v,j} and passed through the image encoder. Classification loss L_CE is the average cross-entropy over all G groups. At inference, per-group logits are averaged (Eq. 6). - Diversity loss L_div minimizes intra-client cosine similarity between group-wise features in both text and visual modalities (Eq. 4), forcing each group to specialize in distinct semantic aspects. Total loss: L = L_CE + λ·L_div. - Dynamic prompt aggregation: each client computes cosine similarity between its G local prompt groups and the server's top-s global prompts from the previous round (Eq. 7). A softmax with temperature converts these into selection probabilities (Eq. 8), and s groups are sampled and uploaded. The server aggregates sampled groups via weighted FedAvg (Eq. 9). Total local capacity is fixed (redistributed, not added) so communication parameters stay at 5.1k regardless of G.

Key Contributions

  • Multi-group paired text-visual prompt architecture for federated VLM adaptation, capturing complementary semantic and instance-level cues without increasing total parameter count.
  • Diversity loss that provably encourages representational separation across prompt groups within each client.
  • Similarity-guided probabilistic sampling for server aggregation, balancing global knowledge retention with client-specific exploration.
  • Lowest communication cost (5.1k parameters) among all FPL methods, with theoretical analysis showing the aggregation strategy reinforces shared semantics while suppressing client-specific noise.

Results

  • Base-to-novel generalization (9 datasets, pathological non-IID): FedMGP achieves 81.85% Combined Metric (CM = (Local + HM)/2), outperforming FedPGP (77.42%), PromptFL (71.31%), PromptFolio (70.29%), FedTPG (70.66%), and FedOTP (57.10%).
  • Label distribution shift (Dirichlet α=0.5, 100 clients): CIFAR-10: 95.48% vs. FedOTP 94.73% (next best); CIFAR-100: 75.39% vs. FedOTP 75.15%.
  • Parameter efficiency: 5.1k communication parameters vs. 8.2k (PromptFL/FedOTP), 16.4k (FedPGP/PromptFolio), 4208.1k (FedTPG), while achieving the highest CM (88.34 in Table 3).
  • Ablation (G=5 groups, Top-s=2, l=2 prompt length, λ=1 diversity weight): removing diversity loss drops CM by 1.8%; text-only drops CM from 85.43 to 84.46; vision-only drops to 75.30.
  • Few-shot: FedMGP underperforms at 1-shot but surpasses all baselines at 2+ shots.

Limitations

  • Underperforms competing methods in the extreme 1-shot regime, as insufficient samples prevent effective disentanglement of shared vs. client-specific knowledge across prompt groups.
  • Evaluated only on CLIP (ViT-B/16); generalization to other VLM architectures is not demonstrated.
  • Domain generalization results are deferred to appendix rather than included in the main table.
  • The diversity loss formulation penalizes all inter-group similarity equally, without accounting for class-conditional structure.

Relevance to Vision-Language Models

FedMGP directly addresses how to efficiently adapt CLIP-style VLMs in privacy-preserving distributed settings—a realistic deployment scenario that is underexplored relative to centralized prompt-tuning work like CoCoOp and ProGrad. By extending prompt learning to multimodal (text+visual) multi-group representations with a principled aggregation strategy, it demonstrates that parameter-efficient VLM adaptation can be made robust to severe data heterogeneity without sacrificing communication efficiency. The finding that visual prompts achieve substantially higher inter-group diversity than text prompts (near-zero vs. 0.5–0.8 intra-client similarity) is a practically useful insight for the broader VLM prompt learning community. This work complements centralized multi-modal prompt tuning research by grounding it in the federated constraints of real deployment.