Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates

• URL: http://arxiv.org/abs/2105.04153v1
• Date: Mon, 10 May 2021 07:15:49 GMT
• Title: Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
• Authors: Laizhong Cui and Xiaoxin Su and Yipeng Zhou and Yi Pan
• Abstract summary: Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS.
• Score: 12.660500431713336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. However, a major obstacle that impedes the wide deployment of FL lies in massive communication traffic. To train high dimensional machine learning models (such as CNN models), heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server (PS), implying that the network resource can be easily exhausted. Compressing model updates is an effective way to reduce the traffic amount. However, a flexible unbiased compression algorithm applicable for both uplink and downlink compression in FL is still absent from existing works. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS. In MUCSC, it is only necessary to transmit cluster centroids and the cluster ID of each model update. Moreover, we prove that: 1) The compressed model updates are unbiased estimation of their original values so that the convergence rate by transmitting compressed model updates is unchanged; 2) MUCSC can guarantee that the influence of the compression error on the model accuracy is minimized. Then, we further propose the boosted MUCSC (B-MUCSC) algorithm, a biased compression algorithm that can achieve an extremely high compression rate by grouping insignificant model updates into a super cluster. B-MUCSC is suitable for scenarios with very scarce network resource. Ultimately, we conduct extensive experiments with the CIFAR-10 and FEMNIST datasets to demonstrate that our algorithms can not only substantially reduce the volume of communication traffic in FL, but also improve the training efficiency in practical networks.

Related papers

• High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates [50.406127962933915]
  We develop solutions to problems which enable us to learn a communication-efficient distributed logistic regression model. In our experiments we demonstrate a large improvement in accuracy over distributed algorithms with only a few distributed update steps needed.
  arXiv  Detail & Related papers  (2024-07-08T19:34:39Z)
• Efficient Model Compression for Hierarchical Federated Learning [10.37403547348343]
  Federated learning (FL) has garnered significant attention due to its capacity to preserve privacy within distributed learning systems. This paper introduces a novel hierarchical FL framework that integrates the benefits of clustered FL and model compression.
  arXiv  Detail & Related papers  (2024-05-27T12:17:47Z)
• Communication-Efficient Federated Learning with Adaptive Compression under Dynamic Bandwidth [6.300376113680886]
  Federated learning can train models without directly providing local data to the server. Recent scholars have achieved the communication efficiency of federated learning mainly by model compression. We show the performance of AdapComFL algorithm, and compare it with existing algorithms.
  arXiv  Detail & Related papers  (2024-05-06T08:00:43Z)
• Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression [28.688895217988925]
  We propose Federated Learning in-network Aggregation with Compression (FediAC) algorithm, consisting of two phases: client voting and model aggregating. FediAC consumes much less memory space and communication traffic than existing works because the first phase can guarantee consensus compression across clients. We conduct extensive experiments by using public datasets to demonstrate that FediAC remarkably surpasses the state-of-the-art baselines in terms of model accuracy and communication traffic.
  arXiv  Detail & Related papers  (2024-02-06T09:00:05Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation [10.541541376305245]
  Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices. FL is hindered by excessive communication costs due to repeated server-client communication during training. We propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation.
  arXiv  Detail & Related papers  (2024-01-25T14:49:15Z)
• Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
  We take a closer theoretical look at Independent Subnetwork Training (IST) IST is a recently proposed and highly effective technique for solving the aforementioned problems. We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
  arXiv  Detail & Related papers  (2023-06-28T18:14:22Z)
• Comfetch: Federated Learning of Large Networks on Constrained Clients via Sketching [28.990067638230254]
  Federated learning (FL) is a popular paradigm for private and collaborative model training on the edge. We propose a novel algorithm, Comdirectional, which allows clients to train large networks using representations of the global neural network.
  arXiv  Detail & Related papers  (2021-09-17T04:48:42Z)
• Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism [56.78673028601739]
  We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT reduces communication by at least $100times$ and $20times$ during DP and MP, respectively. It improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.
  arXiv  Detail & Related papers  (2020-10-18T01:44:42Z)
• Over-the-Air Federated Learning from Heterogeneous Data [107.05618009955094]
  Federated learning (FL) is a framework for distributed learning of centralized models. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local gradient descent (SGD) FL algorithm. We numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.
  arXiv  Detail & Related papers  (2020-09-27T08:28:25Z)
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning [62.440827696638664]
  We introduce a simple algorithm that directly compresses the model differences between neighboring workers. Inspired by the PowerSGD for centralized deep learning, this algorithm uses power steps to maximize the information transferred per bit.
  arXiv  Detail & Related papers  (2020-08-04T09:14:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.