PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees

• URL: http://arxiv.org/abs/2302.06637v3
• Date: Tue, 23 Jul 2024 11:38:37 GMT
• Title: PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees
• Authors: Chulin Xie, De-An Huang, Wenda Chu, Daguang Xu, Chaowei Xiao, Bo Li, Anima Anandkumar,
• Abstract summary: Existing pFL methods introduce high communication and computation costs or are vulnerable to test communication. In PerAda, a parameter distillation and pFL pFL has superior performance, especially under test-time distribution. Our code is available at https://github.com/NV/PerAda.
• Score: 95.87604231887353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized Federated Learning (pFL) has emerged as a promising solution to tackle data heterogeneity across clients in FL. However, existing pFL methods either (1) introduce high communication and computation costs or (2) overfit to local data, which can be limited in scope, and are vulnerable to evolved test samples with natural shifts. In this paper, we propose PerAda, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts. PerAda reduces the costs by leveraging the power of pretrained models and only updates and communicates a small number of additional parameters from adapters. PerAda has good generalization since it regularizes each client's personalized adapter with a global adapter, while the global adapter uses knowledge distillation to aggregate generalized information from all clients. Theoretically, we provide generalization bounds to explain why PerAda improves generalization, and we prove its convergence to stationary points under non-convex settings. Empirically, PerAda demonstrates competitive personalized performance (+4.85% on CheXpert) and enables better out-of-distribution generalization (+5.23% on CIFAR-10-C) on different datasets across natural and medical domains compared with baselines, while only updating 12.6% of parameters per model based on the adapter. Our code is available at https://github.com/NVlabs/PerAda.

Related papers

• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private. We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning [7.005516664708197]
  We propose a novel framework called FedPIA that improves upon the naive combinations of FL and PEFT. We conduct over 2000 client-level experiments utilizing 48 medical image datasets across five different medical vision-language FL task settings. Our experiments demonstrate that FedPIA consistently outperforms the state-of-the-art PEFT-FL baselines.
  arXiv  Detail & Related papers  (2024-12-19T00:24:00Z)
• FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization [19.328216705039527]
  FedMCP is a novel parameter-efficient fine-tuning method with model-contrastive personalization for FL. We show that FedMCP achieves substantial performance improvements over state-of-the-art FL fine-tuning approaches for PLMs.
  arXiv  Detail & Related papers  (2024-08-28T04:19:47Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. To optimize the pruning process itself, only thresholds are communicated between a server and clients instead of parameters. Global thresholds are used to update model parameters by extracting aggregated parameter importance.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Personalized Federated Learning on Long-Tailed Data via Adversarial Feature Augmentation [24.679535905451758]
  PFL aims to learn personalized models for each client based on the knowledge across all clients in a privacy-preserving manner. Existing PFL methods assume that the underlying global data across all clients are uniformly distributed without considering the long-tail distribution. We propose Federated Learning with Adversarial Feature Augmentation (FedAFA) to address this joint problem in PFL.
  arXiv  Detail & Related papers  (2023-03-27T13:00:20Z)
• FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning [18.763298147996238]
  Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. FL faces two critical challenges that hinder its actual performance: data distribution Heterogeneous and high resource costs. We propose FedCLIP to achieve fast generalization and personalization for CLIP in FL.
  arXiv  Detail & Related papers  (2023-02-27T02:49:06Z)
• Towards Efficient Visual Adaption via Structural Re-parameterization [76.57083043547296]
  We propose a parameter-efficient and computational friendly adapter for giant vision models, called RepAdapter. RepAdapter outperforms full tuning by +7.2% on average and saves up to 25% training time, 20% GPU memory, and 94.6% storage cost of ViT-B/16 on VTAB-1k.
  arXiv  Detail & Related papers  (2023-02-16T06:14:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.