An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning

• URL: http://arxiv.org/abs/2211.16703v1
• Date: Wed, 30 Nov 2022 02:55:21 GMT
• Title: An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning
• Authors: Shaohuai Shi, Qing Yang, Yang Xiang, Shuhan Qi, Xuan Wang
• Abstract summary: We design an efficient split fine-tuning framework for edge and cloud collaborative learning. We compress the intermediate output of a neural network to reduce the communication volume between the edge device and the cloud server. Our framework can reduce the communication traffic by 96 times with little impact on the model accuracy.
• Score: 20.118073642453034
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To enable the pre-trained models to be fine-tuned with local data on edge devices without sharing data with the cloud, we design an efficient split fine-tuning (SFT) framework for edge and cloud collaborative learning. We propose three novel techniques in this framework. First, we propose a matrix decomposition-based method to compress the intermediate output of a neural network to reduce the communication volume between the edge device and the cloud server. Second, we eliminate particular links in the model without affecting the convergence performance in fine-tuning. Third, we implement our system atop PyTorch to allow users to easily extend their existing training scripts to enjoy the efficient edge and cloud collaborative learning. Experiments results on 9 NLP datasets show that our framework can reduce the communication traffic by 96 times with little impact on the model accuracy.

Related papers

• Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation [56.79064699832383]
  We establish a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm in which the edge models only need to perform forward propagation. In our CEMA, to reduce the communication burden, we devise two criteria to exclude unnecessary samples from uploading to the cloud.
  arXiv  Detail & Related papers  (2024-02-27T08:47:19Z)
• ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation [47.35179593006409]
  We propose ECLM, an edge-cloud collaborative learning framework for rapid model adaptation for dynamic edge environments. We show that ECLM significantly improves model performance (e.g., 18.89% accuracy increase) and resource efficiency (e.g. 7.12x communication cost reduction) in adapting models to dynamic edge environments.
  arXiv  Detail & Related papers  (2023-11-18T14:10:09Z)
• Shoggoth: Towards Efficient Edge-Cloud Collaborative Real-Time Video Inference via Adaptive Online Learning [33.16911236522438]
  Shoggoth is an efficient edge-cloud collaborative architecture for boosting inference performance on real-time video of changing scenes. Online knowledge distillation improves the accuracy of models suffering from data drift and offloads the labeling process to the cloud. At the edge, we design adaptive training using small batches to adapt models under limited computing power.
  arXiv  Detail & Related papers  (2023-06-27T09:39:42Z)
• Data Augmentation-free Unsupervised Learning for 3D Point Cloud Understanding [61.30276576646909]
  We propose an augmentation-free unsupervised approach for point clouds to learn transferable point-level features via soft clustering, named SoftClu. We exploit the affiliation of points to their clusters as a proxy to enable self-training through a pseudo-label prediction task.
  arXiv  Detail & Related papers  (2022-10-06T10:18:16Z)
• Receptive Field-based Segmentation for Distributed CNN Inference Acceleration in Collaborative Edge Computing [93.67044879636093]
  We study inference acceleration using distributed convolutional neural networks (CNNs) in collaborative edge computing network. We propose a novel collaborative edge computing using fused-layer parallelization to partition a CNN model into multiple blocks of convolutional layers.
  arXiv  Detail & Related papers  (2022-07-22T18:38:11Z)
• Semi-Decentralized Federated Edge Learning with Data and Device Heterogeneity [6.341508488542275]
  Federated edge learning (FEEL) has attracted much attention as a privacy-preserving paradigm to effectively incorporate the distributed data at the network edge for training deep learning models. In this paper, we investigate a novel framework of FEEL, namely semi-decentralized federated edge learning (SD-FEEL), where multiple edge servers are employed to collectively coordinate a large number of client nodes. By exploiting the low-latency communication among edge servers for efficient model sharing, SD-FEEL can incorporate more training data, while enjoying much lower latency compared with conventional federated learning.
  arXiv  Detail & Related papers  (2021-12-20T03:06:08Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [78.44473677588887]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. It inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)
• Semi-Decentralized Federated Edge Learning for Fast Convergence on Non-IID Data [3.983055670167878]
  Federated edge learning (FEEL) has emerged as an effective alternative to reduce the large communication latency in Cloud-based machine learning solutions. We investigate a novel framework of FEEL, namely semi-decentralized federated edge learning (SD-FEEL) By allowing model aggregation between different edge clusters, SD-FEEL enjoys the benefit of FEEL in reducing training latency and improves the learning performance.
  arXiv  Detail & Related papers  (2021-04-26T16:11:47Z)
• Hierarchical Quantized Federated Learning: Convergence Analysis and System Design [7.481427303081613]
  Federated learning is a collaborative machine to train deep neural networks without clients' private data. Previous works assume one central parameter either at the cloud or at the edge. This paper exploits the advantages of both cloud servers and considers Hierarchical Quantized Federated Learning system.
  arXiv  Detail & Related papers  (2021-03-26T05:48:36Z)
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning [73.82875010696849]
  Machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models. This paper focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation.
  arXiv  Detail & Related papers  (2020-03-10T05:52:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.