Communication and Computation Reduction for Split Learning using Asynchronous Training

• URL: http://arxiv.org/abs/2107.09786v1
• Date: Tue, 20 Jul 2021 22:08:13 GMT
• Title: Communication and Computation Reduction for Split Learning using Asynchronous Training
• Abstract summary: This paper proposes a loss-based asynchronous training scheme that updates the client-side model less frequently and only sends/receives activations/gradients in selected epochs. The privacy of the proposed communication reduction based split learning method is almost the same as traditional split learning.
• Score: 9.461478650926608
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Split learning is a promising privacy-preserving distributed learning scheme that has low computation requirement at the edge device but has the disadvantage of high communication overhead between edge device and server. To reduce the communication overhead, this paper proposes a loss-based asynchronous training scheme that updates the client-side model less frequently and only sends/receives activations/gradients in selected epochs. To further reduce the communication overhead, the activations/gradients are quantized using 8-bit floating point prior to transmission. An added benefit of the proposed communication reduction method is that the computations at the client side are reduced due to reduction in the number of client model updates. Furthermore, the privacy of the proposed communication reduction based split learning method is almost the same as traditional split learning. Simulation results on VGG11, VGG13 and ResNet18 models on CIFAR-10 show that the communication cost is reduced by 1.64x-106.7x and the computations in the client are reduced by 2.86x-32.1x when the accuracy degradation is less than 0.5% for the single-client case. For 5 and 10-client cases, the communication cost reduction is 11.9x and 11.3x on VGG11 for 0.5% loss in accuracy.

Related papers

• Decoupled Split Learning via Auxiliary Loss [11.21293654556267]
  Split learning is a distributed training paradigm where a neural network is partitioned between clients and a server.<n>Traditional split learning relies on end-to-end backpropagation across the client-server split point.<n>We develop a beyond-backpropagation training method for split learning.
  arXiv  Detail & Related papers  (2026-01-27T06:41:42Z)
• LoCo: Low-Bit Communication Adaptor for Large-scale Model Training [63.040522637816906]
  Low-bit communication often degrades training quality due to compression information loss. We propose Low-bit Communication Adaptor (LoCo), which compensates local local GPU nodes before, without compromising quality. Experimental results show that across moving large-scale training model frameworks like Megatron-LM and PyTorchs FSDP, LoCo significantly improves compression communication efficiency.
  arXiv  Detail & Related papers  (2024-07-05T13:01:36Z)
• Federated Hyperdimensional Computing [14.844383542052169]
  Federated learning (FL) enables a loose set of participating clients to collaboratively learn a global model via coordination by a central server. Existing FL approaches rely on complex algorithms with massive models, such as deep neural networks (DNNs) We first propose FedHDC, a federated learning framework based on hyperdimensional computing (HDC)
  arXiv  Detail & Related papers  (2023-12-26T09:24:19Z)
• Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing [50.79602839359522]
  We propose HASTE (Hashing for Tractable Efficiency), a parameter-free and data-free module that acts as a plug-and-play replacement for any regular convolution module. We are able to drastically compress latent feature maps without sacrificing much accuracy by using locality-sensitive hashing (LSH) In particular, we are able to instantly drop 46.72% of FLOPs while only losing 1.25% accuracy by just swapping the convolution modules in a ResNet34 on CIFAR-10 for our HASTE module.
  arXiv  Detail & Related papers  (2023-09-29T13:09:40Z)
• ResFed: Communication Efficient Federated Learning by Transmitting Deep Compressed Residuals [24.13593410107805]
  Federated learning enables cooperative training among massively distributed clients by sharing their learned local model parameters. We introduce a residual-based federated learning framework (ResFed), where residuals rather than model parameters are transmitted in communication networks for training. By employing a common prediction rule, both locally and globally updated models are always fully recoverable in clients and the server.
  arXiv  Detail & Related papers  (2022-12-11T20:34:52Z)
• Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach [54.311495894129585]
  We study the limit of communication cost of model aggregation in distributed learning from a rate-distortion perspective. It is found that the communication gain by exploiting the correlation between worker nodes is significant for SignSGD.
  arXiv  Detail & Related papers  (2022-06-28T13:10:40Z)
• FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients [41.623518032533035]
  In split learning, only a small part of the model is stored and trained on clients while the remaining large part of the model only stays at the servers. This paper addresses this issue by compressing the additional communication using a novel clustering scheme accompanied by a gradient correction method.
  arXiv  Detail & Related papers  (2022-01-28T00:09:53Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [65.68511423300812]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. ProgFed inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation [56.886414139084216]
  Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. We propose a communication efficient federated learning method based on knowledge distillation.
  arXiv  Detail & Related papers  (2021-08-30T15:39:54Z)
• Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning [75.45968495410047]
  Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update. We propose an adaptive quantization strategy called AdaFL that aims to achieve communication efficiency as well as a low error floor.
  arXiv  Detail & Related papers  (2021-02-08T19:14:21Z)
• Time-Correlated Sparsification for Communication-Efficient Federated Learning [6.746400031322727]
  Federated learning (FL) enables multiple clients to collaboratively train a shared model without disclosing their local datasets. We introduce a novel time-correlated sparsification scheme, which seeks a certain correlation between the sparse representations used at consecutive iterations in FL. We show that TCS can achieve centralized training accuracy with 100 times sparsification, and up to 2000 times reduction in the communication load when employed together with quantization.
  arXiv  Detail & Related papers  (2021-01-21T20:15:55Z)
• Distributed Sparse SGD with Majority Voting [5.32836690371986]
  We introduce a majority voting based sparse communication strategy for distributed learning. We show that it is possible to achieve up to x4000 compression without any loss in the test accuracy.
  arXiv  Detail & Related papers  (2020-11-12T17:06:36Z)
• Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism [56.78673028601739]
  We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT reduces communication by at least $100times$ and $20times$ during DP and MP, respectively. It improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.
  arXiv  Detail & Related papers  (2020-10-18T01:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.