Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping

• URL: http://arxiv.org/abs/2111.02949v1
• Date: Thu, 4 Nov 2021 15:36:25 GMT
• Title: Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
• Authors: Junya Chen, Sijia Wang, Lawrence Carin, Chenyang Tao
• Abstract summary: We present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization. Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants.
• Score: 77.53019031244908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distributed learning has become an integral tool for scaling up machine learning and addressing the growing need for data privacy. Although more robust to the network topology, decentralized learning schemes have not gained the same level of popularity as their centralized counterparts for being less competitive performance-wise. In this work, we attribute this issue to the lack of synchronization among decentralized learning workers, showing both empirically and theoretically that the convergence rate is tied to the synchronization level among the workers. Such motivated, we present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization. We provide a careful analysis of its convergence and discuss its merits for modern distributed optimization applications, such as deep neural networks. Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants that accommodate asynchronous and randomized communications. To validate the effectiveness of our proposal, we benchmark NGO against competing solutions through an extensive set of tests, with encouraging results reported.

Related papers

• DRACO: Decentralized Asynchronous Federated Learning over Continuous Row-Stochastic Network Matrices [7.389425875982468]
  We propose DRACO, a novel method for decentralized asynchronous Descent (SGD) over row-stochastic gossip wireless networks. Our approach enables edge devices within decentralized networks to perform local training and model exchanging along a continuous timeline. Our numerical experiments corroborate the efficacy of the proposed technique.
  arXiv  Detail & Related papers  (2024-06-19T13:17:28Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [94.2860766709971]
  We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents. Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
  arXiv  Detail & Related papers  (2024-04-04T06:24:11Z)
• Networked Communication for Decentralised Agents in Mean-Field Games [59.01527054553122]
  We introduce networked communication to the mean-field game framework. We show that our architecture has sample guarantees bounded between those of the centralised- and independent-learning cases. We additionally show that the networked approach has significant advantages, over both the centralised and independent alternatives.
  arXiv  Detail & Related papers  (2023-06-05T10:45:39Z)
• Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks [8.860889476382594]
  Decentralized optimization with time-varying networks is an emerging paradigm in machine learning. It saves remarkable communication overhead in large-scale deep training and is more robust in wireless scenarios especially when nodes are moving.
  arXiv  Detail & Related papers  (2022-11-01T15:37:54Z)
• Communication-Efficient Adaptive Federated Learning [17.721884358895686]
  Federated learning is a machine learning paradigm that enables clients to jointly train models without sharing their own localized data. The implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead. We propose a novel communication-efficient adaptive learning method (FedCAMS) with theoretical convergence guarantees.
  arXiv  Detail & Related papers  (2022-05-05T15:47:04Z)
• Asynchronous Upper Confidence Bound Algorithms for Federated Linear Bandits [35.47147821038291]
  We propose a general framework with asynchronous model update and communication for a collection of homogeneous clients and heterogeneous clients. Rigorous theoretical analysis is provided about the regret and communication cost under this distributed learning framework.
  arXiv  Detail & Related papers  (2021-10-04T14:01:32Z)
• Decentralized Personalized Federated Learning for Min-Max Problems [79.61785798152529]
  This paper is the first to study PFL for saddle point problems encompassing a broader range of optimization problems. We propose new algorithms to address this problem and provide a theoretical analysis of the smooth (strongly) convex-(strongly) concave saddle point problems. Numerical experiments for bilinear problems and neural networks with adversarial noise demonstrate the effectiveness of the proposed methods.
  arXiv  Detail & Related papers  (2021-06-14T10:36:25Z)
• Decentralized Statistical Inference with Unrolled Graph Neural Networks [26.025935320024665]
  We propose a learning-based framework, which unrolls decentralized optimization algorithms into graph neural networks (GNNs) By minimizing the recovery error via end-to-end training, this learning-based framework resolves the model mismatch issue. Our convergence analysis reveals that the learned model parameters may accelerate the convergence and reduce the recovery error to a large extent.
  arXiv  Detail & Related papers  (2021-04-04T07:52:34Z)
• Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity [57.275753974812666]
  Federated learning involves learning from data samples distributed across a network of clients while the data remains local. In this paper, we propose a novel straggler-resilient federated learning method that incorporates statistical characteristics of the clients' data to adaptively select the clients in order to speed up the learning procedure.
  arXiv  Detail & Related papers  (2020-12-28T19:21:14Z)
• A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning [49.15799302636519]
  We design a low complexity decentralized learning algorithm to train a recently proposed large neural network in distributed processing nodes (workers) In our setup, the training data is distributed among the workers but is not shared in the training process due to privacy and security concerns. We show that it is possible to achieve equivalent learning performance as if the data is available in a single place.
  arXiv  Detail & Related papers  (2020-09-29T13:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.