Towards Faster Decentralized Stochastic Optimization with Communication Compression

• URL: http://arxiv.org/abs/2405.20114v2
• Date: Mon, 25 Nov 2024 09:00:40 GMT
• Title: Towards Faster Decentralized Stochastic Optimization with Communication Compression
• Authors: Rustem Islamov, Yuan Gao, Sebastian U. Stich,
• Abstract summary: We introduce MoTEF, a novel approach that integrates communication with Momentum Tracking and Error Feedback. Our analysis demonstrates that MoTEF integrates most of the desired properties, and significantly existing methods under data.
• Score: 27.484212303346816
• License:
• Abstract: Communication efficiency has garnered significant attention as it is considered the main bottleneck for large-scale decentralized Machine Learning applications in distributed and federated settings. In this regime, clients are restricted to transmitting small amounts of quantized information to their neighbors over a communication graph. Numerous endeavors have been made to address this challenging problem by developing algorithms with compressed communication for decentralized non-convex optimization problems. Despite considerable efforts, the current results suffer from various issues such as non-scalability with the number of clients, requirements for large batches, or bounded gradient assumption. In this paper, we introduce MoTEF, a novel approach that integrates communication compression with Momentum Tracking and Error Feedback. Our analysis demonstrates that MoTEF achieves most of the desired properties, and significantly outperforms existing methods under arbitrary data heterogeneity. We provide numerical experiments to validate our theoretical findings and confirm the practical superiority of MoTEF.

Related papers

• FedRTS: Federated Robust Pruning via Combinatorial Thompson Sampling [12.067872131025231]
  Federated Learning (FL) enables collaborative model training across distributed clients without data sharing. Current methods use dynamic pruning to improve efficiency by periodically adjusting sparse model topologies while maintaining sparsity. We propose Federated Robust pruning via Thompson Sampling (FedRTS), a novel framework designed to develop robust sparse models.
  arXiv  Detail & Related papers  (2025-01-31T13:26:22Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models. Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity [55.03958223190181]
  We propose the first theoretically grounded accelerated algorithms utilizing unbiased and biased compression under data similarity. Our results are of record and confirmed by experiments on different average losses and datasets.
  arXiv  Detail & Related papers  (2024-12-21T00:40:58Z)
• Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis [50.18156030818883]
  Anomaly and missing data constitute a thorny problem in industrial applications. Deep learning enabled anomaly detection has emerged as a critical direction. The data collected in edge devices contain user privacy.
  arXiv  Detail & Related papers  (2024-11-06T15:38:31Z)
• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration [66.43954501171292]
  We introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. DFedCata consists of two main components: the Moreau envelope function, which addresses parameter inconsistencies, and Nesterov's extrapolation step, which accelerates the aggregation phase. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions.
  arXiv  Detail & Related papers  (2024-10-09T06:17:16Z)
• Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity [72.85248553787538]
  This paper focuses on performance analysis and optimization for wireless FL, considering data heterogeneity, combined with wireless resource allocation. We formulate the loss function minimization problem, under constraints on long-term energy consumption and latency, and jointly optimize client scheduling, resource allocation, and the number of local training epochs (CRE) Experiments on real-world datasets demonstrate that the proposed algorithm outperforms other benchmarks in terms of the learning accuracy and energy consumption.
  arXiv  Detail & Related papers  (2023-08-04T04:18:01Z)
• Asynchronous Federated Learning with Bidirectional Quantized Communications and Buffered Aggregation [39.057968279167966]
  Asynchronous Federated Learning with Buffered Aggregation (FedBuff) is a state-of-the-art algorithm known for its efficiency and high scalability. We present a new algorithm (QAFeL) with a quantization scheme that establishes a shared "hidden" state between the server and clients to avoid the error propagation caused by direct quantization.
  arXiv  Detail & Related papers  (2023-08-01T03:50:58Z)
• Communication-Efficient Federated Distillation with Active Data Sampling [6.516631577963641]
  Federated learning (FL) is a promising paradigm to enable privacy-preserving deep learning from distributed data. Federated Distillation (FD) is a recently proposed alternative to enable communication-efficient and robust FL. This paper presents a generic meta-algorithm for FD and investigate the influence of key parameters through empirical experiments. We propose a communication-efficient FD algorithm with active data sampling to improve the model performance and reduce the communication overhead.
  arXiv  Detail & Related papers  (2022-03-14T07:50:55Z)
• Sample-based and Feature-based Federated Learning via Mini-batch SSCA [18.11773963976481]
  This paper investigates sample-based and feature-based federated optimization. We show that the proposed algorithms can preserve data privacy through the model aggregation mechanism. We also show that the proposed algorithms converge to Karush-Kuhn-Tucker points of the respective federated optimization problems.
  arXiv  Detail & Related papers  (2021-04-13T08:23:46Z)
• A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free! [72.31332210635524]
  Decentralized optimization methods enable on-device training of machine learning models without a central coordinator. We propose a new randomized first-order method which tackles the communication bottleneck by applying randomized compression operators. We prove that our method can solve the problems without any increase in the number of communications compared to the baseline.
  arXiv  Detail & Related papers  (2020-11-03T13:35:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.