Skip to content

Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DP-FPL is a federated prompt learning framework for vision-language models (specifically CLIP) that simultaneously achieves personalization, generalization, and differential privacy. It decomposes each client's local soft prompt into low-rank components (for generalization) plus a residual term (for expressiveness), applying LDP to the low-rank factors and GDP to the shared global prompt. This selective noise injection significantly reduces the accuracy penalty of differential privacy compared to uniform noise baselines.

Problem

Federated Prompt Learning (FPL) for VLMs faces a three-way tension: (1) personalization via non-IID local prompts risks overfitting, (2) standard DP noise degrades accuracy severely when applied uniformly to full-rank prompts, and (3) the Prompt-as-a-Service paradigm exposes trained prompts to membership inference attacks (MIA). No prior FPL method simultaneously addresses personalization, generalization, and formal differential privacy guarantees.

Method

Each client maintains a global prompt pG,i (aggregated via federated averaging) and a local prompt pL,i (kept locally). At every training round, pL,i is factorized via the power method (one iteration, cost O(kmn) vs. O(m²n) for SVD) into low-rank components u_i, v_i and a residual r_i = pL,i - u_i v_i. The full personalized prompt is pi = pG,i + u_i v_i + r_i.

Privacy is applied selectively: - LDP (local DP, Gaussian noise with σ_L) is added to gradients w.r.t. u_i and v_i only — not to the residual, which is not used in prompt recomputation. - GDP (global DP, Gaussian noise with σ_G) is added to the aggregated global gradient at the server.

The residual compensates for expressiveness lost during low-rank factorization, acting as a counterweight to the regularization-like effect of DP noise. Privacy budget satisfies (ε,δ)-LDP and (ε,δ)-GDP via advanced composition, with σ ∝ S√(T log(1/δ)) / ε.

Key Contributions

  • Low-rank factorization with per-round re-factorization (vs. one-time factorization in FedPGP), plus a residual term to recover expressive capacity lost to rank reduction and DP noise.
  • Asymmetric DP application: LDP on low-rank local components, GDP on the aggregated global prompt — first FPL method combining both DP modes in a single framework.
  • Formal privacy theorem (Theorem 3.3) giving closed-form σ_L, σ_G to achieve target (ε,δ) guarantees under advanced composition.
  • Demonstration that moderate DP noise can improve generalization (acts as regularization), with diminishing returns at very high noise (ε ≤ 0.05).

Results

Evaluated on Caltech101, OxfordPets, OxfordFlowers, Food101 (10 clients, pathological split) and CIFAR-100 (25/50 clients, Dirichlet split) using CLIP backbone; baselines are PromptFL, FedOTP, FedPGP (rank 8).

Local (personalization) accuracy, ε=0.1: - Caltech101: DP-FPL 92.71% vs. FedPGP 80.01%, FedOTP 84.98%, PromptFL 80.42% - OxfordPets: DP-FPL 85.25% vs. FedPGP 63.21%, FedOTP 59.54%, PromptFL 72.77% - OxfordFlowers: DP-FPL 72.11% vs. FedPGP 52.42%, FedOTP 42.31% - Food101: DP-FPL 80.57% vs. FedPGP 75.18%, FedOTP 70.48%

CIFAR-100 (25 clients), ε=0.1: DP-FPL 59.53% vs. FedPGP 55.76%, FedOTP 46.95%, PromptFL 50.86%

  • At ε=0.01 (strictest), DP-FPL still consistently leads all baselines across all datasets.
  • Neighbor (generalization) accuracy shows analogous advantage under DP; MIA success rate falls below 50% (random guess) at ε=0.1 (reported in appendix).
  • Ablation: residual term provides largest gains at low rank (rank 1) and high noise (ε=0.01).

Limitations

  • Residual term r_i receives no DP protection; its privacy implications when the full prompt pi is published are not formally analyzed.
  • Theoretical analysis of the residual's regularization effect is deferred to future work; the regularization interpretation is empirically supported but unproven.
  • Experiments use CLIP as the only VLM backbone; generalization to other multimodal LLMs (e.g., LLaVA, Flamingo) is untested.
  • Assumes honest clients and server; byzantine-robust or malicious-server settings are out of scope.
  • Communication cost analysis is absent; sending global prompt gradients every round has overhead not quantified.

Relevance to Vision-Language Models

This work directly extends prompt tuning for CLIP-based VLMs into federated, privacy-sensitive deployment scenarios, which is increasingly relevant as VLMs move toward real-world use in healthcare and finance. The low-rank factorization strategy draws on the same intrinsic-dimensionality insights that underpin LoRA and soft prompt methods in centralized VLM fine-tuning, connecting federated adaptation to mainstream PEFT research. The Prompt-as-a-Service framing highlights a concrete threat model (MIA on distributed prompts) that the VLM community has largely overlooked. For researchers tracking VLMs, DP-FPL demonstrates a practical path to deploying personalized VLM prompts at scale without centralized data collection.