Skip to content

Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

pFedMoAP (Personalized Federated Mixture of Adaptive Prompts) is a federated prompt learning framework for CLIP-like VLMs that breaks the traditional single-global-model constraint by letting clients download multiple pre-aggregated prompt experts from other clients. A lightweight attention-based gating network dynamically fuses local and non-local text features conditioned on each image, achieving strong personalization under heterogeneous data distributions. It outperforms prior federated prompt learning methods across 9 datasets with both label and feature shifts.

Problem

Existing federated prompt learning methods (PromptFL, pFedPrompt, FedOTP) rigidly follow the traditional FL paradigm where each client downloads only one globally aggregated model. This restriction is unjustified for prompts — which are orders of magnitude smaller than full models (8,192 parameters for a length-16 prompt vs. 25.6M for ResNet50) — and limits knowledge sharing, causing poor personalization under extreme data heterogeneity (non-IID label and feature shifts).

Method

pFedMoAP operates in federated rounds over CLIP (frozen image encoder f(·) and text encoder g(·)):

  1. Local prompt training: Each client i maintains a local prompt P_i initialized from the global aggregated prompt, updated via SGD with cross-entropy loss.
  2. Non-local expert selection: A server-maintained pool stores per-client prompts. At round start, each client receives K non-local prompts selected via KNN (L2 distance in prompt space), treating similar-distribution clients as closer collaborators. Non-local prompts are kept frozen locally.
  3. Attention-based gating network: A client-specific MHA gating network (G(·|θ_i)) takes image feature I_k = f(x_k) as query and concatenated local+non-local text features as key/value. To stay parameter-efficient, features are pooled to dimension d_gating=128 before MHA (66K params vs. 4.2M full-dim). Output is enhanced text feature T_MoE^(c) per class.
  4. Final logits: logit_c = sim(I_k, T_MoE^(c)) + λ · sim(I_k, T_L^(c)), where λ weights the local prompt's direct contribution. Only P_i and θ_i are updated; θ_i stays local, P_i is uploaded to the pool.

Key Contributions

  • Paradigm shift: challenges the single-global-model restriction for federated prompt learning, enabling multi-prompt downloading justified by prompts' lightweight nature.
  • pFedMoAP framework: MoE-based personalized federated prompt learning specifically for CLIP-like VLMs.
  • Attention-based gating network: order-agnostic, variable-K, parameter-efficient, leverages CLIP's cross-modal alignment via scaled dot-product attention; generalizable beyond FL to multi-task/domain-adaptation prompt settings.
  • KNN-based non-local expert assignment on the server that acts as sparse gating at the system level.

Results

  • Flowers102 (pathological non-IID): pFedMoAP 98.41% vs. FedOTP 96.23% (+2.18%) and pFedPrompt 86.46% (+11.95%).
  • OxfordPets: 99.06% vs. FedOTP 98.82%.
  • Food101: 93.39% vs. FedOTP 92.73%.
  • Caltech101: 97.95% vs. FedOTP 97.02%.
  • DTD: 89.13% vs. FedOTP 87.64%.
  • CIFAR10 (Dir(α=0.5), 100 clients): 83.46% vs. CoOp 80.84%, PromptFL 73.29%.
  • CIFAR100: 53.42% vs. CoOp 48.74%, PromptFL 45.00%.
  • DomainNet (feature+label shift): 38.80% average vs. CoOp 36.99%, PromptFL 24.48% (FedAvg hurt performance vs. local training).
  • Office-Caltech10: 38.31% vs. CoOp 32.37%, PromptFL 23.41%.
  • Consistent gains across 1–16 shot settings; performance gap widens with more shots.

Limitations

  • Evaluated only on image classification tasks; applicability to generation or VQA tasks not explored.
  • KNN-based expert selection assumes meaningful L2 distances in prompt space; may degrade if prompt space is not well-structured early in training (first-round participants skip MoE).
  • Communication cost increases with K non-local prompts per round, though still negligible vs. full model transfer; no analysis of bandwidth budget vs. K trade-off.
  • Fixed CLIP backbone assumed — incompatible with settings where the vision or language encoder must also be fine-tuned.
  • Privacy analysis is absent; sharing un-aggregated per-client prompts (as non-local experts) may leak more client-specific information than a global aggregate.

Relevance to Vision-Language Models

This paper directly addresses efficient adaptation of CLIP-like VLMs in privacy-constrained, decentralized settings, a practical deployment scenario for models like CLIP that are often used off-the-shelf. The attention-based gating network repurposes CLIP's cross-modal alignment capability to dynamically weight multiple text experts per image query, offering a reusable module for multi-prompt VLM settings beyond FL (e.g., multi-task or domain-adaptive prompting). For researchers tracking VLMs, pFedMoAP illustrates how the lightweight prompt learning paradigm unlocks qualitatively new system designs that full-model fine-tuning cannot support. It also surfaces the under-studied problem of prompt heterogeneity across data distributions, complementing work on prompt robustness and generalization in VLMs.