LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication

• URL: http://arxiv.org/abs/2604.09970v1
• Date: Sat, 11 Apr 2026 01:02:47 GMT
• Title: LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
• Abstract summary: We propose a unified multiple Local Training (MLT) Decentralized framework with Adam-type updates and Compressed communication (CC)<n>LoDAdaC accommodates a broad class of gradients for its local adaptive updates, including AMSGrad, Adam, and AdaGrad.<n>Experiments on image classification and GPT-style language model training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.
• Score: 16.272293204712888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practical performance in deep learning and centralized distributed settings. However, their convergence properties remain largely unexplored in decentralized settings involving multiple local training steps, such as federated learning. To address this limitation, we propose LoDAdaC, a unified multiple Local Training (MLT) Decentralized framework with Adam-type updates and Compressed communication (CC). LoDAdaC accommodates a broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit quantization and sparsification. MLT and CC enable LoDAdaC to achieve multiplied reduction of communication cost, while the technique of adaptive updates enables fast convergence. We rigorously prove the combined advantage through complexity analysis. In addition, experiments on image classification and GPT-style language model training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.

Related papers

• Event-Triggered Gossip for Distributed Learning [61.70659996356528]
  We develop a new event-triggered gossip framework for distributed learning to reduce inter-node communication.<n>We analyze bf71.61% with only a marginal performance loss, compared with the conventional full-text-of-the-art distributed learning methods.
  arXiv  Detail & Related papers  (2026-02-22T10:13:43Z)
• Accelerating Decentralized Optimization via Overlapping Local Steps [6.713278402701195]
  We propose a novel approach to accelerate decentralized computation without sacrificing theoretical guarantees.<n>We show OLDSGD retains the same iteration as Local Decentralized SGDOLDD while improving per-iteration convergence under different levels of communication delays.
  arXiv  Detail & Related papers  (2026-01-04T11:40:22Z)
• Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization [8.579148218325168]
  dissertation explores strategies to improve communication efficiency, focusing on model compression, local training, and personalization.<n>We establish a unified framework for biased and unbiased compression operators with convergence guarantees.<n>We propose adaptive local training strategies that incorporate personalization to accelerate convergence and mitigate client drift.
  arXiv  Detail & Related papers  (2025-09-10T02:19:56Z)
• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration [66.43954501171292]
  We introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. DFedCata consists of two main components: the Moreau envelope function, which addresses parameter inconsistencies, and Nesterov's extrapolation step, which accelerates the aggregation phase. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions.
  arXiv  Detail & Related papers  (2024-10-09T06:17:16Z)
• Local Methods with Adaptivity via Scaling [38.99428012275441]
  This paper aims to merge the local training technique with the adaptive approach to develop efficient distributed learning methods. We consider the classical Local SGD method and enhance it with a scaling feature. In addition to theoretical analysis, we validate the performance of our methods in practice by training a neural network.
  arXiv  Detail & Related papers  (2024-06-02T19:50:05Z)
• FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models [52.13056951033747]
  Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server.<n>A critical bottleneck in FL is the communication cost.<n>Our work is inspired by the innovative Scaffnew algorithm, which has advanced the reduction of communication complexity in FL.<n>We introduce FedComLoc, integrating practical and effective compression into Scaffnew to further enhance communication efficiency.
  arXiv  Detail & Related papers  (2024-03-14T22:29:59Z)
• LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression [56.01900711954956]
  We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats.<n>LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions.
  arXiv  Detail & Related papers  (2024-03-07T09:22:50Z)
• CoDeC: Communication-Efficient Decentralized Continual Learning [6.663641564969944]
  Training at the edge utilizes continuously evolving data generated at different locations. Privacy concerns prohibit the co-location of this spatially as well as temporally distributed data. We propose CoDeC, a novel communication-efficient decentralized continual learning algorithm.
  arXiv  Detail & Related papers  (2023-03-27T16:52:17Z)
• CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning [62.65937719264881]
  Federated learning facilitates learning across clients without transferring local data on these clients to a central server. We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning. Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
  arXiv  Detail & Related papers  (2020-12-15T12:20:28Z)
• Adaptive Serverless Learning [114.36410688552579]
  We propose a novel adaptive decentralized training approach, which can compute the learning rate from data dynamically. Our theoretical results reveal that the proposed algorithm can achieve linear speedup with respect to the number of workers. To reduce the communication-efficient overhead, we further propose a communication-efficient adaptive decentralized training approach.
  arXiv  Detail & Related papers  (2020-08-24T13:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.