FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning

• URL: http://arxiv.org/abs/2308.09160v1
• Date: Thu, 17 Aug 2023 19:22:30 GMT
• Title: FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning
• Authors: Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, Chen Chen
• Abstract summary: We investigate where and how to partially personalize a Vision Transformers (ViT) model. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix. We evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness compared to several advanced PFL methods.
• Score: 9.950367271170592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized Federated Learning (PFL) represents a promising solution for decentralized learning in heterogeneous data environments. Partial model personalization has been proposed to improve the efficiency of PFL by selectively updating local model parameters instead of aggregating all of them. However, previous work on partial model personalization has mainly focused on Convolutional Neural Networks (CNNs), leaving a gap in understanding how it can be applied to other popular models such as Vision Transformers (ViTs). In this work, we investigate where and how to partially personalize a ViT model. Specifically, we empirically evaluate the sensitivity to data distribution of each type of layer. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix, which leverages plugins to transfer information from the aggregated model to the local client as a personalization. Finally, we evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness in improving the model's performance compared to several advanced PFL methods.

Related papers

• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning [16.344719695572586]
  We propose a novel scheme to inject personalized prior knowledge into a global model in each client. At the heart of our proposed approach is a framework, the PFL with Bregman Divergence (pFedBreD) Our method reaches the state-of-the-art performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks.
  arXiv  Detail & Related papers  (2023-10-13T15:21:25Z)
• PFL-GAN: When Client Heterogeneity Meets Generative Models in Personalized Federated Learning [55.930403371398114]
  We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL) PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation. The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
  arXiv  Detail & Related papers  (2023-08-23T22:38:35Z)
• FedSoup: Improving Generalization and Personalization in Federated Learning via Selective Model Interpolation [32.36334319329364]
  Cross-silo federated learning (FL) enables the development of machine learning models on datasets distributed across data centers. Recent research has found that current FL algorithms face a trade-off between local and global performance when confronted with distribution shifts. We propose a novel federated model soup method to optimize the trade-off between local and global performance.
  arXiv  Detail & Related papers  (2023-07-20T00:07:29Z)
• Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training [67.67045085186797]
  Almost all existing systems have to face large communication burdens if the central FL server fails. It personalizes the "right" in the deep models by alternately updating the shared and personal parameters. To further promote the shared parameters aggregation process, we propose DFed integrating the local Sharpness Miniization.
  arXiv  Detail & Related papers  (2023-05-24T13:52:18Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Visual Prompt Based Personalized Federated Learning [83.04104655903846]
  We propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
  arXiv  Detail & Related papers  (2023-03-15T15:02:15Z)
• Tackling Data Heterogeneity in Federated Learning with Class Prototypes [44.746340839025194]
  We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization. We show that imposing uniformity helps to combat prototype collapse while infusing class semantics improves local models.
  arXiv  Detail & Related papers  (2022-12-06T05:15:38Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Federated Adversarial Training with Transformers [16.149924042225106]
  Federated learning (FL) has emerged to enable global model training over distributed clients' data while preserving its privacy. This paper investigates feasibility with different federated model aggregation methods and different vision transformer models with different tokenization and classification head techniques.
  arXiv  Detail & Related papers  (2022-06-05T09:07:09Z)
• A Closer Look at Personalization in Federated Image Classification [33.27317065917578]
  Federated Learning (FL) is developed to learn a single global model across the decentralized data. This paper shows that it is possible to achieve flexible personalization after the convergence of the global model. We propose RepPer, an independent two-stage personalized FL framework.
  arXiv  Detail & Related papers  (2022-04-22T06:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.