DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology

• URL: http://arxiv.org/abs/2111.05505v1
• Date: Wed, 10 Nov 2021 03:00:40 GMT
• Title: DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology
• Authors: Zhikun Chen, Daofeng Li, Jinkang Zhu and Sihai Zhang
• Abstract summary: Federated learning (FL) is a distributed machine learning framework where a central parameter server coordinates many local users to train a globally consistent model. This paper devises a new DFL implementation coined DACFL, where each user trains its model using its own training data and exchanges the intermediate models with its neighbors. The DACFL treats the progress of each user's local training as a discrete-time process and employs a first order dynamic average consensus (FODAC) method to track the textitaverage model in the absence of the PS.
• Score: 4.234367850767171
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) is a burgeoning distributed machine learning framework where a central parameter server (PS) coordinates many local users to train a globally consistent model. Conventional federated learning inevitably relies on a centralized topology with a PS. As a result, it will paralyze once the PS fails. To alleviate such a single point failure, especially on the PS, some existing work has provided decentralized FL (DFL) implementations like CDSGD and D-PSGD to facilitate FL in a decentralized topology. However, there are still some problems with these methods, e.g., significant divergence between users' final models in CDSGD and a network-wide model average necessity in D-PSGD. In order to solve these deficiency, this paper devises a new DFL implementation coined as DACFL, where each user trains its model using its own training data and exchanges the intermediate models with its neighbors through a symmetric and doubly stochastic matrix. The DACFL treats the progress of each user's local training as a discrete-time process and employs a first order dynamic average consensus (FODAC) method to track the \textit{average model} in the absence of the PS. In this paper, we also provide a theoretical convergence analysis of DACFL on the premise of i.i.d data to strengthen its rationality. The experimental results on MNIST, Fashion-MNIST and CIFAR-10 validate the feasibility of our solution in both time-invariant and time-varying network topologies, and declare that DACFL outperforms D-PSGD and CDSGD in most cases.

Related papers

• Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme [5.5058010121503]
  Decentralized Federated Learning (DFL) has become popular due to its robustness and avoidance of centralized coordination. We propose a novel textitsparse-to-sparser training scheme: DA-DPFL. Our experiments showcase that DA-DPFL substantially outperforms DFL baselines in test accuracy, while achieving up to $5$ times reduction in energy costs.
  arXiv  Detail & Related papers  (2024-04-24T16:03:34Z)
• Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates [71.81037644563217]
  Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning. As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers. We propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion.
  arXiv  Detail & Related papers  (2024-03-27T09:14:36Z)
• Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees [18.24213566328972]
  Decentralized decentralized learning (DFL) captures FL settings where both (i) model updates and (ii) model aggregations are carried out by the clients without a central server. DSpodFL consistently achieves speeds compared with baselines under various system settings.
  arXiv  Detail & Related papers  (2024-02-05T19:02:19Z)
• Scheduling and Communication Schemes for Decentralized Federated Learning [0.31410859223862103]
  A decentralized federated learning (DFL) model with the gradient descent (SGD) algorithm has been introduced. Three scheduling policies for DFL have been proposed for communications between the clients and the parallel servers. Results show that the proposed scheduling polices have an impact both on the speed of convergence and in the final global model.
  arXiv  Detail & Related papers  (2023-11-27T17:35:28Z)
• DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning [52.83811558753284]
  Decentralized learning (DFL) discards the central server and establishes a decentralized communication network. Existing DFL methods still suffer from two major challenges: local inconsistency and local overfitting.
  arXiv  Detail & Related papers  (2023-08-16T11:22:36Z)
• Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training [67.67045085186797]
  Almost all existing systems have to face large communication burdens if the central FL server fails. It personalizes the "right" in the deep models by alternately updating the shared and personal parameters. To further promote the shared parameters aggregation process, we propose DFed integrating the local Sharpness Miniization.
  arXiv  Detail & Related papers  (2023-05-24T13:52:18Z)
• Event-Triggered Decentralized Federated Learning over Resource-Constrained Edge Devices [12.513477328344255]
  Federated learning (FL) is a technique for distributed machine learning (ML) In traditional FL algorithms, trained models at the edge are periodically sent to a central server for aggregation. We develop a novel methodology for fully decentralized FL, where devices conduct model aggregation via cooperative consensus formation.
  arXiv  Detail & Related papers  (2022-11-23T00:04:05Z)
• Decentralized Event-Triggered Federated Learning with Heterogeneous Communication Thresholds [12.513477328344255]
  We propose a novel methodology for distributed model aggregations via asynchronous, event-triggered consensus iterations over a network graph topology. We demonstrate that our methodology achieves the globally optimal learning model under standard assumptions in distributed learning and graph consensus literature.
  arXiv  Detail & Related papers  (2022-04-07T20:35:37Z)
• Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point [51.47520726446029]
  cooperative edge learning (CE-FL) is a distributed machine learning architecture. We model the processes taken during CE-FL, and conduct analytical training. We show the effectiveness of our framework with the data collected from a real-world testbed.
  arXiv  Detail & Related papers  (2022-03-26T00:41:57Z)
• Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks [50.68446003616802]
  Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices. We develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning.
  arXiv  Detail & Related papers  (2022-02-07T05:11:01Z)
• Consensus Control for Decentralized Deep Learning [72.50487751271069]
  Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop.
  arXiv  Detail & Related papers  (2021-02-09T13:58:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.