FedSPU: Personalized Federated Learning for Resource-constrained Devices with Stochastic Parameter Update

• URL: http://arxiv.org/abs/2403.11464v1
• Date: Mon, 18 Mar 2024 04:31:38 GMT
• Title: FedSPU: Personalized Federated Learning for Resource-constrained Devices with Stochastic Parameter Update
• Authors: Ziru Niu, Hai Dong, A. K. Qin,
• Abstract summary: Federated Dropout has emerged as a popular strategy to address this challenge. We propose federated learning with parameter update (FedSPU) Experimental results demonstrate that FedSPU outperforms federated dropout by 7.57% on average in terms of accuracy.
• Score: 0.27309692684728615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized Federated Learning (PFL) is widely employed in IoT applications to handle high-volume, non-iid client data while ensuring data privacy. However, heterogeneous edge devices owned by clients may impose varying degrees of resource constraints, causing computation and communication bottlenecks for PFL. Federated Dropout has emerged as a popular strategy to address this challenge, wherein only a subset of the global model, i.e. a \textit{sub-model}, is trained on a client's device, thereby reducing computation and communication overheads. Nevertheless, the dropout-based model-pruning strategy may introduce bias, particularly towards non-iid local data. When biased sub-models absorb highly divergent parameters from other clients, performance degradation becomes inevitable. In response, we propose federated learning with stochastic parameter update (FedSPU). Unlike dropout that tailors the global model to small-size local sub-models, FedSPU maintains the full model architecture on each device but randomly freezes a certain percentage of neurons in the local model during training while updating the remaining neurons. This approach ensures that a portion of the local model remains personalized, thereby enhancing the model's robustness against biased parameters from other clients. Experimental results demonstrate that FedSPU outperforms federated dropout by 7.57\% on average in terms of accuracy. Furthermore, an introduced early stopping scheme leads to a significant reduction of the training time by \(24.8\%\sim70.4\%\) while maintaining high accuracy.

Related papers

• FedImpro: Measuring and Improving Client Update in Federated Learning [77.68805026788836]
  Federated Learning (FL) models often experience client drift caused by heterogeneous data. We present an alternative perspective on client drift and aim to mitigate it by generating improved local models.
  arXiv  Detail & Related papers  (2024-02-10T18:14:57Z)
• Mitigating System Bias in Resource Constrained Asynchronous Federated Learning Systems [2.8790600498444032]
  We propose a dynamic global model aggregation method within Asynchronous Federated Learning (AFL) deployments. Our method scores and adjusts the weighting of client model updates based on their upload frequency to accommodate differences in device capabilities.
  arXiv  Detail & Related papers  (2024-01-24T10:51:15Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Efficient Personalized Federated Learning via Sparse Model-Adaptation [47.088124462925684]
  Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. We propose pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. We show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T12:21:34Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Personalized Federated Learning via Variational Bayesian Inference [6.671486716769351]
  Federated learning faces huge challenges from model overfitting due to the lack of data and statistical diversity among clients. This paper proposes a novel personalized federated learning method via Bayesian variational inference named pFedBayes. Experiments show that the proposed method outperforms other advanced personalized methods on personalized models.
  arXiv  Detail & Related papers  (2022-06-16T07:37:02Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• FedPrune: Towards Inclusive Federated Learning [1.308951527147782]
  Federated learning (FL) is a distributed learning technique that trains a shared model over distributed data in a privacy-preserving manner. We propose FedPrune; a system that tackles this challenge by pruning the global model for slow clients based on their device characteristics. By using insights from Central Limit Theorem, FedPrune incorporates a new aggregation technique that achieves robust performance over non-IID data.
  arXiv  Detail & Related papers  (2021-10-27T06:33:38Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.