Rate-Constrained Quantization for Communication-Efficient Federated Learning

• URL: http://arxiv.org/abs/2409.06319v1
• Date: Tue, 10 Sep 2024 08:22:01 GMT
• Title: Rate-Constrained Quantization for Communication-Efficient Federated Learning
• Authors: Shayan Mohajer Hamidi, Ali Bereyhi,
• Abstract summary: We develop a novel quantized FL framework, called textbfrate-textbfconstrained textbffederated learning (RC-FED) We formulate this scheme, as a joint optimization in which the quantization distortion is minimized while the rate of encoded gradients is kept below a target threshold. We analyze the convergence behavior of RC-FED, and show its superior performance against baseline quantized FL schemes on several datasets.
• Score: 5.632231145349047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantization is a common approach to mitigate the communication cost of federated learning (FL). In practice, the quantized local parameters are further encoded via an entropy coding technique, such as Huffman coding, for efficient data compression. In this case, the exact communication overhead is determined by the bit rate of the encoded gradients. Recognizing this fact, this work deviates from the existing approaches in the literature and develops a novel quantized FL framework, called \textbf{r}ate-\textbf{c}onstrained \textbf{fed}erated learning (RC-FED), in which the gradients are quantized subject to both fidelity and data rate constraints. We formulate this scheme, as a joint optimization in which the quantization distortion is minimized while the rate of encoded gradients is kept below a target threshold. This enables for a tunable trade-off between quantization distortion and communication cost. We analyze the convergence behavior of RC-FED, and show its superior performance against baseline quantized FL schemes on several datasets.

Related papers

• Green, Quantized Federated Learning over Wireless Networks: An Energy-Efficient Design [68.86220939532373]
  The finite precision level is captured through the use of quantized neural networks (QNNs) that quantize weights and activations in fixed-precision format. The proposed FL framework can reduce energy consumption until convergence by up to 70% compared to a baseline FL algorithm.
  arXiv  Detail & Related papers  (2022-07-19T16:37:24Z)
• Learning Representations for CSI Adaptive Quantization and Feedback [51.14360605938647]
  We propose an efficient method for adaptive quantization and feedback in frequency division duplexing systems. Existing works mainly focus on the implementation of autoencoder (AE) neural networks for CSI compression. We recommend two different methods: one based on a post training quantization and the second one in which the codebook is found during the training of the AE.
  arXiv  Detail & Related papers  (2022-07-13T08:52:13Z)
• Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach [54.311495894129585]
  We study the limit of communication cost of model aggregation in distributed learning from a rate-distortion perspective. It is found that the communication gain by exploiting the correlation between worker nodes is significant for SignSGD.
  arXiv  Detail & Related papers  (2022-06-28T13:10:40Z)
• How to Attain Communication-Efficient DNN Training? Convert, Compress, Correct [19.440030100380632]
  This paper introduces CO3 -- an algorithm for communication-efficient federated Deep Neural Network (DNN) training. CO3 takes its name from three processing applied which reduce the communication load when transmitting the local DNN gradients from the remote users to the Server.
  arXiv  Detail & Related papers  (2022-04-18T08:22:55Z)
• Wireless Quantized Federated Learning: A Joint Computation and Communication Design [36.35684767732552]
  In this paper, we aim to minimize the total convergence time of FL, by quantizing the local model parameters prior to uplink transmission. We jointly optimize the computing, communication resources and number of quantization bits, in order to guarantee minimized convergence time across all global rounds.
  arXiv  Detail & Related papers  (2022-03-11T12:30:08Z)
• Communication-Efficient Federated Learning via Quantized Compressed Sensing [82.10695943017907]
  The presented framework consists of gradient compression for wireless devices and gradient reconstruction for a parameter server. Thanks to gradient sparsification and quantization, our strategy can achieve a higher compression ratio than one-bit gradient compression. We demonstrate that the framework achieves almost identical performance with the case that performs no compression.
  arXiv  Detail & Related papers  (2021-11-30T02:13:54Z)
• FedDQ: Communication-Efficient Federated Learning with Descending Quantization [5.881154276623056]
  Federated learning (FL) is an emerging privacy-preserving distributed learning scheme. FL suffers from critical communication bottleneck due to large model size and frequent model aggregation. This paper proposes an opposite approach to do adaptive quantization.
  arXiv  Detail & Related papers  (2021-10-05T18:56:28Z)
• DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning [22.83609192604322]
  We propose a novel dynamically quantized SGD (DQ-SGD) framework to dynamically adjust the quantization scheme for each gradient descent step. We show that our quantization scheme achieves better tradeoffs between the communication cost and learning performance than other state-of-the-art gradient quantization methods.
  arXiv  Detail & Related papers  (2021-07-30T12:22:31Z)
• Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning [75.45968495410047]
  Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update. We propose an adaptive quantization strategy called AdaFL that aims to achieve communication efficiency as well as a low error floor.
  arXiv  Detail & Related papers  (2021-02-08T19:14:21Z)
• Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training [99.42912552638168]
  Communication of gradients is costly for training deep neural networks with multiple devices in computer vision applications. In this work, we deduce the optimal condition of both the binary and multi-level gradient quantization for textbfANY gradient distribution. Based on the optimal condition, we develop two novel quantization schemes: biased BinGrad and unbiased ORQ for binary and multi-level gradient quantization respectively.
  arXiv  Detail & Related papers  (2020-02-25T18:28:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.