Skip to content

pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; personalized federated fine-tuning for CLIP with multi-modal adapters

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

pFedMMA introduces a personalized federated learning framework for adapting CLIP-style VLMs using multi-modal adapters, where each adapter has modality-specific up/down projections and a single shared cross-modal projection. During training, all adapter components are updated locally, but only the shared projection is aggregated on the server. This asymmetric scheme achieves state-of-the-art balance between local personalization and generalization to unseen classes across eleven datasets.

Problem

Federated prompt tuning methods (FedOTP, FedPGP, pFedMoAP) achieve strong local personalization under heterogeneous data but collapse on novel/unseen classes — FedOTP, for instance, reaches 97.34% local accuracy yet only 36.69 HM averaged over seven datasets in the 16-shot setting. Meanwhile, multi-modal adapters, which are known to outperform prompt tuning in centralized few-shot settings, have not been adapted to personalized federated learning.

Method

pFedMMA inserts parallel multi-modal adapters (MMAs) into the upper \(L - \ell + 1\) transformer blocks of both CLIP's vision and text encoders, leaving lower layers frozen to preserve transferable representations. Each adapter at block \(j\) for modality \(o \in \{I, T\}\) computes:

\[A_j^{(o)}(z_j^{(o)}) = W_{ju}^{(o)} \cdot \delta\!\left(W_{js} \cdot \delta\!\left(W_{jd}^{(o)} \cdot z_j^{(o)}\right)\right)\]

where \(W_{jd}^{(o)} \in \mathbb{R}^{r \times d}\) and \(W_{ju}^{(o)} \in \mathbb{R}^{d \times r}\) are modality-specific down/up projections (\(r \ll d\)), and \(W_{js}\) is a shared projection matrix used by both modalities. The parallel adapter adds its output to the frozen backbone output with a scalar \(\alpha\).

Training is asymmetric: every client updates all components locally for \(E\) epochs via cross-entropy SGD, but only \(W_{js,i}^{t,E}\) is uploaded. The server aggregates it as \(W_{js}^{t+1} = \sum_{i=1}^{N} p_i W_{js,i}^{t,E}\) (\(p_i = n_i / n\)). Modality-specific up/down projections remain permanently local, acting as client-specific "personalized" components.

Key Contributions

  • Multi-modal adapter design with a shared cross-modal projection \(W_{js}\) that is the only globally communicated parameter, achieving communication efficiency.
  • Asymmetric PFL training scheme: personalized up/down projections learned locally, global generalization enforced through shared projection aggregation.
  • Selective top-layer insertion strategy that reduces trainable parameter count while maintaining cross-modal alignment at the most discriminative layers.
  • Comprehensive evaluation on 11 datasets spanning both label-shift (pathological non-IID class splits on 7 CLIP datasets, CIFAR-10/100 with Dirichlet partitioning) and feature-shift (DomainNet 6 domains, Office-Caltech10 4 domains) scenarios.

Results

All results are 16-shot with ViT-B/16 unless noted; HM = harmonic mean over local, base, and novel class accuracy.

  • Average over 7 CLIP datasets (16-shot): pFedMMA HM 84.15% vs. pFedMoAP 71.05% (+13.10 pp), FedOTP 36.69%, zero-shot CLIP 81.21%.
  • SUN397 HM: pFedMMA 79.34% vs. pFedMoAP 42.41%, FedPGP 65.02%.
  • OxfordPets HM: pFedMMA 79.79% vs. pFedMoAP 60.28%, FedOTP 34.65%.
  • Caltech101 (second table, ViT-B/16): pFedMMA HM 94.78% vs. pFedMoAP 88.87%.
  • UCF101 (second table): pFedMMA HM 96.88% vs. pFedMoAP 95.37%.
  • pFedMMA is the only method to consistently improve novel-class accuracy while retaining competitive local/base accuracy; FedOTP sacrifices novel generalization for local gains.
  • Radar chart results (Fig. 1) show pFedMMA dominates all shot counts (1–16) in average HM.

Limitations

  • Shared projection aggregation uses FedAvg with uniform weighting by dataset size, inheriting FedAvg's known sensitivity to extreme heterogeneity; no alternative aggregation (e.g., robust or momentum-based) is explored.
  • Architecture assumes a frozen transformer backbone (CLIP ViT-B/16 or ViT-B/32); applicability to decoder-based or instruction-tuned VLMs is not demonstrated.
  • The top-layer adapter insertion threshold \(\ell\) is a hyperparameter borrowed from Yang et al. (2024); ablation on \(\ell\) for federated settings is deferred to appendix without discussion in the main text.
  • Text is truncated before DomainNet and CIFAR numerical results are presented; full feature-shift performance cannot be assessed from the provided excerpt.
  • Communication cost analysis is relegated to Appendix E with no main-text numbers reported.

Relevance to Vision-Language Models

pFedMMA directly addresses the practical deployment gap for CLIP-style VLMs in privacy-sensitive, heterogeneous real-world settings (healthcare, legal, industrial) where centralized fine-tuning is infeasible. The key insight — that the shared cross-modal projection in a multi-modal adapter is the right locus for global aggregation, while modality-specific projections capture local distribution shift — is a principled contribution to understanding what to share vs. personalize in VLM adaptation. The base-to-novel HM evaluation metric is the standard VLM generalization benchmark, so results are directly comparable to the centralized CLIP adaptation literature. This work establishes multi-modal adapters as a superior alternative to prompt tuning in the federated regime, which has implications for on-device or edge deployments of foundation VLMs.