Accelerating Optimization and Machine Learning through Decentralization

• URL: http://arxiv.org/abs/2604.19518v1
• Date: Tue, 21 Apr 2026 14:40:07 GMT
• Title: Accelerating Optimization and Machine Learning through Decentralization
• Abstract summary: Decentralized optimization enables multiple devices to learn a global machine learning model while each individual device only has access to its local dataset.<n>We show that decentralization can paradoxically accelerate convergence, outperforming centralized methods in the number of iterations needed to reach optimal solutions.
• Score: 3.325688379088159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decentralized optimization enables multiple devices to learn a global machine learning model while each individual device only has access to its local dataset. By avoiding the need for training data to leave individual users' devices, it enhances privacy and scalability compared to conventional centralized learning, where all data has to be aggregated to a central server. However, decentralized optimization has traditionally been viewed as a necessary compromise, used only when centralized processing is impractical due to communication constraints or data privacy concerns. In this study, we show that decentralization can paradoxically accelerate convergence, outperforming centralized methods in the number of iterations needed to reach optimal solutions. Through examples in logistic regression and neural network training, we demonstrate that distributing data and computation across multiple agents can lead to faster learning than centralized approaches, even when each iteration is assumed to take the same amount of time, whether performed centrally on the full dataset or decentrally on local subsets. This finding challenges longstanding assumptions and reveals decentralization as a strategic advantage, offering new opportunities for more efficient optimization and machine learning.

Related papers

• Towards Serverless Semi-Decentralized Federated Learning with Heterogeneous Optimizers [49.91939413642465]
  We investigate cluster formation, involving number and composition of clusters, in decentralized federated learning (FL)<n>We propose serverless semi-decentralized FL (SSD-FL), a methodology requiring no persistent server infrastructure.<n>SSD-FL segments global rounds into intra-cluster and inter-cluster regimes, ensuring global convergence and consensus.
  arXiv  Detail & Related papers  (2026-06-04T20:05:18Z)
• Guided Model Merging for Hybrid Data Learning: Leveraging Centralized Data to Refine Decentralized Models [29.605620036963924]
  Current network training paradigms primarily focus on either centralized or decentralized data regimes.<n>We propose a novel framework that constructs a model atlas from decentralized models and leverages centralized data.<n>Our method synergizes federated learning (to exploit decentralized data) and model merging (to utilize centralized data), enabling effective training under hybrid data availability.
  arXiv  Detail & Related papers  (2025-03-26T01:00:35Z)
• RESIST: Resilient Decentralized Learning Using Consensus Gradient Descent [11.22833419439317]
  Empirical robustness risk (ERM) is a cornerstone of modern machine learning (ML)<n>This paper focuses on the man-in-the-middle (MITM) attack, which can cause models to deviate significantly from their intended ERM solutions.<n>We propose RESIST, an algorithm designed to be robust against adversarially compromised communication links.
  arXiv  Detail & Related papers  (2025-02-11T21:48:10Z)
• DRACO: Decentralized Asynchronous Federated Learning over Row-Stochastic Wireless Networks [7.389425875982468]
  We propose DRACO, a novel method for decentralized asynchronous Descent (SGD) over row-stochastic gossip wireless networks.<n>Our approach enables edge devices within decentralized networks to perform local training and model exchanging along a continuous timeline.<n>Our numerical experiments corroborate the efficacy of the proposed technique.
  arXiv  Detail & Related papers  (2024-06-19T13:17:28Z)
• Asynchronous Parallel Incremental Block-Coordinate Descent for Decentralized Machine Learning [55.198301429316125]
  Machine learning (ML) is a key technique for big-data-driven modelling and analysis of massive Internet of Things (IoT) based intelligent and ubiquitous computing. For fast-increasing applications and data amounts, distributed learning is a promising emerging paradigm since it is often impractical or inefficient to share/aggregate data. This paper studies the problem of training an ML model over decentralized systems, where data are distributed over many user devices.
  arXiv  Detail & Related papers  (2022-02-07T15:04:15Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• Consensus Control for Decentralized Deep Learning [72.50487751271069]
  Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop.
  arXiv  Detail & Related papers  (2021-02-09T13:58:33Z)
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data [77.88594632644347]
  Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge. We propose a novel momentum-based method to mitigate this decentralized training difficulty.
  arXiv  Detail & Related papers  (2021-02-09T11:27:14Z)
• Optimising cost vs accuracy of decentralised analytics in fog computing environments [0.4898659895355355]
  Data gravity, a fundamental concept in Fog Computing, points towards decentralisation of computation for data analysis. We propose an analytical framework able to find the optimal operating point in this continuum. We show through simulations that the model accurately predicts the optimal trade-off, quite often an emphintermediate point between full centralisation and full decentralisation.
  arXiv  Detail & Related papers  (2020-12-09T19:05:44Z)
• Toward Multiple Federated Learning Services Resource Sharing in Mobile Edge Networks [88.15736037284408]
  We study a new model of multiple federated learning services at the multi-access edge computing server. We propose a joint resource optimization and hyper-learning rate control problem, namely MS-FEDL. Our simulation results demonstrate the convergence performance of our proposed algorithms.
  arXiv  Detail & Related papers  (2020-11-25T01:29:41Z)
• Adaptive Serverless Learning [114.36410688552579]
  We propose a novel adaptive decentralized training approach, which can compute the learning rate from data dynamically. Our theoretical results reveal that the proposed algorithm can achieve linear speedup with respect to the number of workers. To reduce the communication-efficient overhead, we further propose a communication-efficient adaptive decentralized training approach.
  arXiv  Detail & Related papers  (2020-08-24T13:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.