Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization

• URL: http://arxiv.org/abs/2202.11453v1
• Date: Wed, 23 Feb 2022 12:07:02 GMT
• Title: Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization
• Authors: Jaehong Yoon, Geon Park, Wonyong Jeong, Sung Ju Hwang
• Abstract summary: We introduce a pragmatic Federated Learning scenario with bitwidth Heterogeneous Federated Learning (BHFL) BHFL brings in a new challenge, that the aggregation of model parameters with different bitwidths could result in severe performance degeneration. We propose ProWD framework, which has a trainable weight dequantizer at the central server that progressively reconstructs the low-bitwidth weights into higher bitwidth weights, and finally into full-precision weights.
• Score: 58.31288475660333
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In practical federated learning scenarios, the participating devices may have different bitwidths for computation and memory storage by design. However, despite the progress made in device-heterogeneous federated learning scenarios, the heterogeneity in the bitwidth specifications in the hardware has been mostly overlooked. We introduce a pragmatic FL scenario with bitwidth heterogeneity across the participating devices, dubbed as Bitwidth Heterogeneous Federated Learning (BHFL). BHFL brings in a new challenge, that the aggregation of model parameters with different bitwidths could result in severe performance degeneration, especially for high-bitwidth models. To tackle this problem, we propose ProWD framework, which has a trainable weight dequantizer at the central server that progressively reconstructs the low-bitwidth weights into higher bitwidth weights, and finally into full-precision weights. ProWD further selectively aggregates the model parameters to maximize the compatibility across bit-heterogeneous weights. We validate ProWD against relevant FL baselines on the benchmark datasets, using clients with varying bitwidths. Our ProWD largely outperforms the baseline FL algorithms as well as naive approaches (e.g. grouped averaging) under the proposed BHFL scenario.

Related papers

• PIPE : Parallelized Inference Through Post-Training Quantization Ensembling of Residual Expansions [23.1120983784623]
  PIPE is a quantization method that leverages residual error expansion, along with group sparsity and an ensemble approximation for better parallelization. It achieves superior performance on every benchmarked application (from vision to NLP tasks), architecture (ConvNets, transformers) and bit-width.
  arXiv  Detail & Related papers  (2023-11-27T13:29:34Z)
• NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients [44.89061671579694]
  Federated learning (FL) enables distributed training while preserving data privacy, but stragglers-slow or incapable clients-can significantly slow down the total training time and degrade performance. We propose nested federated learning (NeFL), a framework that efficiently divides deep neural networks into submodels using both depthwise and widthwise scaling. NeFL achieves performance gain, especially for the worst-case submodel compared to baseline approaches.
  arXiv  Detail & Related papers  (2023-08-15T13:29:14Z)
• Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices [18.1568276196989]
  Federated Learning (FL) is a paradigm to distributively learn machine learning models from decentralized data that remains on-device. We introduce multiple variants of federated averaging algorithm that train neural networks robust to quantization. Our results demonstrate that integrating quantization robustness results in FL models that are significantly more robust to different bit-widths during quantized on-device inference.
  arXiv  Detail & Related papers  (2022-06-22T05:11:44Z)
• SlimFL: Federated Learning with Superposition Coding over Slimmable Neural Networks [56.68149211499535]
  Federated learning (FL) is a key enabler for efficient communication and computing leveraging devices' distributed computing capabilities. This paper proposes a novel learning framework by integrating FL and width-adjustable slimmable neural networks (SNNs) We propose a communication and energy-efficient SNN-based FL (named SlimFL) that jointly utilizes superposition coding (SC) for global model aggregation and superposition training (ST) for updating local models.
  arXiv  Detail & Related papers  (2022-03-26T15:06:13Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)
• Feature Distillation Interaction Weighting Network for Lightweight Image Super-Resolution [25.50790871331823]
  We propose a lightweight yet efficient Feature Distillation Interaction Weighted Network (FDIWN) FDIWN is superior to other models to strike a good balance between model performance and efficiency.
  arXiv  Detail & Related papers  (2021-12-16T06:20:35Z)
• Low-Latency Federated Learning over Wireless Channels with Differential Privacy [142.5983499872664]
  In federated learning (FL), model training is distributed over clients and local models are aggregated by a central server. In this paper, we aim to minimize FL training delay over wireless channels, constrained by overall training performance as well as each client's differential privacy (DP) requirement.
  arXiv  Detail & Related papers  (2021-06-20T13:51:18Z)
• Bayesian Federated Learning over Wireless Networks [87.37301441859925]
  Federated learning is a privacy-preserving and distributed training method using heterogeneous data sets stored at local devices. This paper presents an efficient modified BFL algorithm called scalableBFL (SBFL)
  arXiv  Detail & Related papers  (2020-12-31T07:32:44Z)
• Wide-band butterfly network: stable and efficient inversion via multi-frequency neural networks [1.2891210250935143]
  We introduce an end-to-end deep learning architecture called the wide-band butterfly network (WideBNet) for approximating the inverse scattering map from wide-band scattering data. This architecture incorporates tools from computational harmonic analysis, such as the butterfly factorization, and traditional multi-scale methods, such as the Cooley-Tukey FFT algorithm.
  arXiv  Detail & Related papers  (2020-11-24T21:48:43Z)
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels [69.18343801164741]
  partitioned edge learning (PARTEL) implements parameter-server training, a well known distributed learning method, in wireless network. We consider the case of deep neural network (DNN) models which can be trained using PARTEL by introducing some auxiliary variables.
  arXiv  Detail & Related papers  (2020-10-08T15:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.