Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme

• URL: http://arxiv.org/abs/2404.15943v3
• Date: Mon, 22 Jul 2024 21:58:05 GMT
• Title: Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme
• Authors: Qianyu Long, Qiyuan Wang, Christos Anagnostopoulos, Daning Bi,
• Abstract summary: Decentralized Federated Learning (DFL) has become popular due to its robustness and avoidance of centralized coordination. We propose a novel textitsparse-to-sparser training scheme: DA-DPFL. Our experiments showcase that DA-DPFL substantially outperforms DFL baselines in test accuracy, while achieving up to $5$ times reduction in energy costs.
• Score: 5.5058010121503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decentralized Federated Learning (DFL) has become popular due to its robustness and avoidance of centralized coordination. In this paradigm, clients actively engage in training by exchanging models with their networked neighbors. However, DFL introduces increased costs in terms of training and communication. Existing methods focus on minimizing communication often overlooking training efficiency and data heterogeneity. To address this gap, we propose a novel \textit{sparse-to-sparser} training scheme: DA-DPFL. DA-DPFL initializes with a subset of model parameters, which progressively reduces during training via \textit{dynamic aggregation} and leads to substantial energy savings while retaining adequate information during critical learning periods. Our experiments showcase that DA-DPFL substantially outperforms DFL baselines in test accuracy, while achieving up to $5$ times reduction in energy costs. We provide a theoretical analysis of DA-DPFL's convergence by solidifying its applicability in decentralized and personalized learning. The code is available at:https://github.com/EricLoong/da-dpfl

Related papers

• FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion [48.90879664138855]
  One-shot Federated Learning (OFL) significantly reduces communication costs in FL by aggregating trained models only once. However, the performance of advanced OFL methods is far behind the normal FL. We propose a novel learning approach to endow OFL with superb performance and low communication and storage costs, termed as FuseFL.
  arXiv  Detail & Related papers  (2024-10-27T09:07:10Z)
• R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Models [83.77114091471822]
  Split federated learning (SFL) is a compute-efficient paradigm in distributed machine learning (ML) A challenge in SFL, particularly when deployed over wireless channels, is the susceptibility of transmitted model parameters to adversarial jamming. This is particularly pronounced for word embedding parameters in large language models (LLMs), which are crucial for language understanding. A physical layer framework is developed for resilient SFL with LLMs (R-SFLLM) over wireless networks.
  arXiv  Detail & Related papers  (2024-07-16T12:21:29Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Adaptive Decentralized Federated Learning in Energy and Latency Constrained Wireless Networks [4.03161352925235]
  In Federated Learning (FL), with parameter aggregated by a central node, the communication overhead is a substantial concern. Recent studies have introduced Decentralized Federated Learning (DFL) as a viable alternative. We formulate a problem that minimizes the loss function of DFL while considering energy and latency constraints.
  arXiv  Detail & Related papers  (2024-03-29T09:17:40Z)
• Decentralized Federated Learning: A Survey and Perspective [45.81975053649379]
  Decentralized FL (DFL) is a decentralized network architecture that eliminates the need for a central server. DFL enables direct communication between clients, resulting in significant savings in communication resources.
  arXiv  Detail & Related papers  (2023-06-02T15:12:58Z)
• Hierarchical Personalized Federated Learning Over Massive Mobile Edge Computing Networks [95.39148209543175]
  We propose hierarchical PFL (HPFL), an algorithm for deploying PFL over massive MEC networks. HPFL combines the objectives of training loss minimization and round latency minimization while jointly determining the optimal bandwidth allocation.
  arXiv  Detail & Related papers  (2023-03-19T06:00:05Z)
• How Much Does It Cost to Train a Machine Learning Model over Distributed Data Sources? [4.222078489059043]
  Federated learning allows devices to train a machine learning model without sharing their raw data. Server-less FL approaches like gossip federated learning (GFL) and blockchain-enabled federated learning (BFL) have been proposed to mitigate these issues. GFL is able to save the 18% of training time, the 68% of energy and the 51% of data to be shared with respect to the CFL solution, but it is not able to reach the level of accuracy of CFL. BFL represents a viable solution for implementing decentralized learning with a higher level of security, at the cost of an extra energy usage and data sharing
  arXiv  Detail & Related papers  (2022-09-15T08:13:40Z)
• Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
  Federated learning (FL) is vulnerable to heterogeneously distributed data. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
  arXiv  Detail & Related papers  (2022-01-27T08:43:11Z)
• DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology [4.234367850767171]
  Federated learning (FL) is a distributed machine learning framework where a central parameter server coordinates many local users to train a globally consistent model. This paper devises a new DFL implementation coined DACFL, where each user trains its model using its own training data and exchanges the intermediate models with its neighbors. The DACFL treats the progress of each user's local training as a discrete-time process and employs a first order dynamic average consensus (FODAC) method to track the textitaverage model in the absence of the PS.
  arXiv  Detail & Related papers  (2021-11-10T03:00:40Z)
• Decentralized Federated Learning: Balancing Communication and Computing Costs [21.694468026280806]
  Decentralized federated learning (DFL) is a powerful framework of distributed machine learning. We propose a general decentralized federated learning framework to strike a balance between communication-efficiency and convergence performance. Experiment results based on MNIST and CIFAR-10 datasets illustrate the superiority of DFL over traditional decentralized SGD methods.
  arXiv  Detail & Related papers  (2021-07-26T09:09:45Z)
• A Framework for Energy and Carbon Footprint Analysis of Distributed and Federated Edge Learning [48.63610479916003]
  This article breaks down and analyzes the main factors that influence the environmental footprint of distributed learning policies. It models both vanilla and decentralized FL policies driven by consensus. Results show that FL allows remarkable end-to-end energy savings (30%-40%) for wireless systems characterized by low bit/Joule efficiency.
  arXiv  Detail & Related papers  (2021-03-18T16:04:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.