Skip to content

NormFit: A Lightweight Solution for Few-Shot Federated Learning with Non-IID Data

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NormFit is a few-shot federated learning method for VLMs that fine-tunes only the Pre-LayerNorm parameters (~1.5K params) of CLIP's vision encoder, simultaneously achieving state-of-the-art accuracy and dramatic reductions in communication and compute cost under non-IID data. It outperforms prior FL-VLM methods (FedCLIP, FedTPG, FedDAT, FLoRA) across 10 diverse datasets while transmitting only 4.1 KB per user per round.

Problem

Adapting pre-trained VLMs (e.g., CLIP) in federated learning suffers from a triple bottleneck: (1) high communication overhead from transmitting large parameter updates, (2) heavy local computation per round, and (3) degraded accuracy under non-IID user data where class imbalance causes prompt/adapter methods to underfit minority classes. Existing FL-VLM methods improve one or two of these axes but fail to address all three simultaneously.

Method

NormFit freezes all CLIP parameters except the Pre-LayerNorm of the vision encoder — the single normalization layer applied to patch embeddings before any transformer block. With only ~1.5K trainable parameters (scale γ and shift β for the embedding dimension), each FL user back-propagates on these weights using few labeled images per class and standard text prompts ("a photo of a [class]"), then transmits the update to a central server for aggregation.

The design is motivated by two theoretical arguments: - Generalization gap (Theorem 1): For CLIP ViT-B/32 (N=12, d=768), fine-tuning all LayerNorms yields a generalization gap up to ~30× larger than fine-tuning only the Pre-LayerNorm (εall/εpre ≲ 29.98), because the number of trainable parameters directly scales the gap bound. - Lipschitz influence (Theorem 2): The Pre-LayerNorm at k=1 has the largest amplification factor A₁ = ∏ⱼ₌₂ᴺ Lⱼ for its direct influence on the global Lipschitz bound, and its cascading influence propagates through all N−1 subsequent layers — more than any deeper LayerNorm. This makes it the point of maximal leverage for global model regularization with minimal parameter change.

The vision encoder is chosen over the text encoder because visual features exhibit greater domain variability (lighting, scale, background), while CLIP's text encoder provides stable cross-domain semantics.

Key Contributions

  • NormFit method: Fine-tune only Pre-LayerNorm of CLIP's vision encoder in few-shot FL; no architectural changes, no external modules.
  • Theoretical justification: Proves a generalization gap ratio bound (Theorem 1) and a Lipschitz influence analysis (Theorem 2) establishing the Pre-LayerNorm as optimal.
  • State-of-the-art accuracy: Outperforms representative and recent FL-VLM baselines on 10 datasets under non-IID (Dirichlet β=0.1) conditions.
  • Efficiency: 1.5K trainable params, 4.1 KB communication, 85 GFLOPs/user — orders of magnitude below all baselines.
  • Versatility: Functions as standalone or as a plug-in add-on to existing methods (CoOp, TIP-Adapter) boosting their performance without modification.

Results

  • Average accuracy (10 datasets, non-IID β=0.1): NormFit 73.72% vs. FLoRA 72.64%, FedTPG 70.37%, FedCLIP 69.33%, FedDAT 70.11%, CoCoOp 64.71%, CoOp 62.08%, TIP-Adapter 70.81%.
  • Communication cost: 4.1 KB/user vs. FLoRA 68 KB, FedTPG 870 KB, FedCLIP 1,500 KB; full fine-tuning 4.1×10⁵ KB.
  • Computation cost: 85 GFLOPs/user vs. CoOp 230, FLoRA 2.8×10⁵, FedDAT 8.5×10⁵, full fine-tuning 8.4×10⁵.
  • As add-on: CoOp+NormFit 73.90% (+11.82 pp over CoOp alone); TIP-Adapter+NormFit 73.27% (+2.46 pp).
  • Zero-shot robustness: NormFit better preserves zero-shot accuracy on out-of-training datasets than fine-tuning all LayerNorms (e.g., when trained on Food, CIFAR-10 accuracy: NormFit 83.92% vs. all-LN 76.53%).
  • Ablation: Progressively unfreezing more LayerNorms beyond Pre-LayerNorm monotonically degrades average accuracy, confirming minimal is optimal.

Limitations

  • May underperform under extreme domain shifts where deeper feature recalibration is necessary.
  • Applies a uniform fine-tuning strategy across all users; no personalization for heterogeneous data quality or domain diversity.
  • In high-data regimes (many samples per class), other methods may surpass NormFit.
  • Evaluated only with CLIP as the VLM backbone; generalizability to other VLM architectures not demonstrated.

Relevance to Vision-Language Models

NormFit directly targets the deployment gap for CLIP-style VLMs in privacy-constrained, resource-limited environments, showing that the Pre-LayerNorm of the vision encoder is a uniquely efficient adaptation handle — both empirically and theoretically. The finding that excessive fine-tuning of deeper layers degrades CLIP's vision-text alignment has broader implications for parameter-efficient fine-tuning (PEFT) design in VLMs, complementing work like LoRA and adapter tuning. For VLM researchers, the theoretical result quantifying generalization gap scaling with parameter count provides a principled basis for choosing which layers to fine-tune in low-data regimes. The work also establishes that vision-side adaptation dominates text-side adaptation in few-shot imbalanced settings, a practically useful design heuristic.