FedDQ: Communication-Efficient Federated Learning with Descending Quantization

• URL: http://arxiv.org/abs/2110.02291v1
• Date: Tue, 5 Oct 2021 18:56:28 GMT
• Title: FedDQ: Communication-Efficient Federated Learning with Descending Quantization
• Authors: Linping Qu, Shenghui Song, Chi-Ying Tsui
• Abstract summary: Federated learning (FL) is an emerging privacy-preserving distributed learning scheme. FL suffers from critical communication bottleneck due to large model size and frequent model aggregation. This paper proposes an opposite approach to do adaptive quantization.
• Score: 5.881154276623056
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) is an emerging privacy-preserving distributed learning scheme. Due to the large model size and frequent model aggregation, FL suffers from critical communication bottleneck. Many techniques have been proposed to reduce the communication volume, including model compression and quantization, where quantization with increasing number of levels has been proposed. This paper proposes an opposite approach to do adaptive quantization. First, we present the drawback of ascending-trend quantization based on the characteristics of training. Second, we formulate the quantization optimization problem and theoretical analysis shows that quantization with decreasing number of levels is preferred. Then we propose two strategies to guide the adaptive quantization process by using the change in training loss and the range of model update. Experimental results on three sets of benchmarks show that descending-trend quantization not only saves more communication bits but also helps FL converge faster, when compares with current ascending-trend quantization.

Related papers

• Rate-Constrained Quantization for Communication-Efficient Federated Learning [5.632231145349047]
  We develop a novel quantized FL framework, called textbfrate-textbfconstrained textbffederated learning (RC-FED) We formulate this scheme, as a joint optimization in which the quantization distortion is minimized while the rate of encoded gradients is kept below a target threshold. We analyze the convergence behavior of RC-FED, and show its superior performance against baseline quantized FL schemes on several datasets.
  arXiv  Detail & Related papers  (2024-09-10T08:22:01Z)
• FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization [11.673528138087244]
  Federated learning (FL) is a powerful machine learning paradigm which leverages the data as well as the computational resources of clients, while protecting clients' data privacy. Previous research has primarily focused on the uplink communication, employing either fixed-bit quantization or adaptive quantization methods. In this work, we introduce a holistic approach by joint uplink and downlink adaptive quantization to reduce the communication overhead.
  arXiv  Detail & Related papers  (2024-06-26T08:14:23Z)
• Performance Optimization for Variable Bitwidth Federated Learning in Wireless Networks [103.22651843174471]
  This paper considers improving wireless communication and computation efficiency in federated learning (FL) via model quantization. In the proposed bitwidth FL scheme, edge devices train and transmit quantized versions of their local FL model parameters to a coordinating server, which aggregates them into a quantized global model and synchronizes the devices. We show that the FL training process can be described as a Markov decision process and propose a model-based reinforcement learning (RL) method to optimize action selection over iterations.
  arXiv  Detail & Related papers  (2022-09-21T08:52:51Z)
• Learning Representations for CSI Adaptive Quantization and Feedback [51.14360605938647]
  We propose an efficient method for adaptive quantization and feedback in frequency division duplexing systems. Existing works mainly focus on the implementation of autoencoder (AE) neural networks for CSI compression. We recommend two different methods: one based on a post training quantization and the second one in which the codebook is found during the training of the AE.
  arXiv  Detail & Related papers  (2022-07-13T08:52:13Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss [61.26793005355441]
  Cluster-Promoting Quantization (CPQ) finds the optimal quantization grids for neural networks. DropBits is a new bit-drop technique that revises the standard dropout regularization to randomly drop bits instead of neurons. We experimentally validate our method on various benchmark datasets and network architectures.
  arXiv  Detail & Related papers  (2021-09-05T15:15:07Z)
• In-Hindsight Quantization Range Estimation for Quantized Training [5.65658124285176]
  We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training.
  arXiv  Detail & Related papers  (2021-05-10T10:25:28Z)
• Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning [75.45968495410047]
  Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update. We propose an adaptive quantization strategy called AdaFL that aims to achieve communication efficiency as well as a low error floor.
  arXiv  Detail & Related papers  (2021-02-08T19:14:21Z)
• Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training [99.42912552638168]
  Communication of gradients is costly for training deep neural networks with multiple devices in computer vision applications. In this work, we deduce the optimal condition of both the binary and multi-level gradient quantization for textbfANY gradient distribution. Based on the optimal condition, we develop two novel quantization schemes: biased BinGrad and unbiased ORQ for binary and multi-level gradient quantization respectively.
  arXiv  Detail & Related papers  (2020-02-25T18:28:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.