Finite-Time Consensus Learning for Decentralized Optimization with
Nonlinear Gossiping
- URL: http://arxiv.org/abs/2111.02949v1
- Date: Thu, 4 Nov 2021 15:36:25 GMT
- Title: Finite-Time Consensus Learning for Decentralized Optimization with
Nonlinear Gossiping
- Authors: Junya Chen, Sijia Wang, Lawrence Carin, Chenyang Tao
- Abstract summary: We present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization.
Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants.
- Score: 77.53019031244908
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Distributed learning has become an integral tool for scaling up machine
learning and addressing the growing need for data privacy. Although more robust
to the network topology, decentralized learning schemes have not gained the
same level of popularity as their centralized counterparts for being less
competitive performance-wise. In this work, we attribute this issue to the lack
of synchronization among decentralized learning workers, showing both
empirically and theoretically that the convergence rate is tied to the
synchronization level among the workers. Such motivated, we present a novel
decentralized learning framework based on nonlinear gossiping (NGO), that
enjoys an appealing finite-time consensus property to achieve better
synchronization. We provide a careful analysis of its convergence and discuss
its merits for modern distributed optimization applications, such as deep
neural networks. Our analysis on how communication delay and randomized chats
affect learning further enables the derivation of practical variants that
accommodate asynchronous and randomized communications. To validate the
effectiveness of our proposal, we benchmark NGO against competing solutions
through an extensive set of tests, with encouraging results reported.
Related papers
- DRACO: Decentralized Asynchronous Federated Learning over Continuous Row-Stochastic Network Matrices [7.389425875982468]
We propose DRACO, a novel method for decentralized asynchronous Descent (SGD) over row-stochastic gossip wireless networks.
Our approach enables edge devices within decentralized networks to perform local training and model exchanging along a continuous timeline.
Our numerical experiments corroborate the efficacy of the proposed technique.
arXiv Detail & Related papers (2024-06-19T13:17:28Z) - Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [94.2860766709971]
We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents.
Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
arXiv Detail & Related papers (2024-04-04T06:24:11Z) - Networked Communication for Decentralised Agents in Mean-Field Games [59.01527054553122]
We introduce networked communication to the mean-field game framework.
We show that our architecture has sample guarantees bounded between those of the centralised- and independent-learning cases.
We additionally show that the networked approach has significant advantages, over both the centralised and independent alternatives.
arXiv Detail & Related papers (2023-06-05T10:45:39Z) - Optimal Complexity in Non-Convex Decentralized Learning over
Time-Varying Networks [8.860889476382594]
Decentralized optimization with time-varying networks is an emerging paradigm in machine learning.
It saves remarkable communication overhead in large-scale deep training and is more robust in wireless scenarios especially when nodes are moving.
arXiv Detail & Related papers (2022-11-01T15:37:54Z) - Communication-Efficient Adaptive Federated Learning [17.721884358895686]
Federated learning is a machine learning paradigm that enables clients to jointly train models without sharing their own localized data.
The implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead.
We propose a novel communication-efficient adaptive learning method (FedCAMS) with theoretical convergence guarantees.
arXiv Detail & Related papers (2022-05-05T15:47:04Z) - Asynchronous Upper Confidence Bound Algorithms for Federated Linear
Bandits [35.47147821038291]
We propose a general framework with asynchronous model update and communication for a collection of homogeneous clients and heterogeneous clients.
Rigorous theoretical analysis is provided about the regret and communication cost under this distributed learning framework.
arXiv Detail & Related papers (2021-10-04T14:01:32Z) - Decentralized Personalized Federated Learning for Min-Max Problems [79.61785798152529]
This paper is the first to study PFL for saddle point problems encompassing a broader range of optimization problems.
We propose new algorithms to address this problem and provide a theoretical analysis of the smooth (strongly) convex-(strongly) concave saddle point problems.
Numerical experiments for bilinear problems and neural networks with adversarial noise demonstrate the effectiveness of the proposed methods.
arXiv Detail & Related papers (2021-06-14T10:36:25Z) - Decentralized Statistical Inference with Unrolled Graph Neural Networks [26.025935320024665]
We propose a learning-based framework, which unrolls decentralized optimization algorithms into graph neural networks (GNNs)
By minimizing the recovery error via end-to-end training, this learning-based framework resolves the model mismatch issue.
Our convergence analysis reveals that the learned model parameters may accelerate the convergence and reduce the recovery error to a large extent.
arXiv Detail & Related papers (2021-04-04T07:52:34Z) - Straggler-Resilient Federated Learning: Leveraging the Interplay Between
Statistical Accuracy and System Heterogeneity [57.275753974812666]
Federated learning involves learning from data samples distributed across a network of clients while the data remains local.
In this paper, we propose a novel straggler-resilient federated learning method that incorporates statistical characteristics of the clients' data to adaptively select the clients in order to speed up the learning procedure.
arXiv Detail & Related papers (2020-12-28T19:21:14Z) - A Low Complexity Decentralized Neural Net with Centralized Equivalence
using Layer-wise Learning [49.15799302636519]
We design a low complexity decentralized learning algorithm to train a recently proposed large neural network in distributed processing nodes (workers)
In our setup, the training data is distributed among the workers but is not shared in the training process due to privacy and security concerns.
We show that it is possible to achieve equivalent learning performance as if the data is available in a single place.
arXiv Detail & Related papers (2020-09-29T13:08:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.