Communication and Storage Efficient Federated Split Learning

• URL: http://arxiv.org/abs/2302.05599v1
• Date: Sat, 11 Feb 2023 04:44:29 GMT
• Title: Communication and Storage Efficient Federated Split Learning
• Authors: Yujia Mu, Cong Shen
• Abstract summary: Federated Split Learning preserves the parallel model training principle of FL. Server has to maintain separate models for every client, resulting in a significant computation and storage requirement. This paper proposes a communication and storage efficient federated and split learning strategy.
• Score: 19.369076939064904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) is a popular distributed machine learning (ML) paradigm, but is often limited by significant communication costs and edge device computation capabilities. Federated Split Learning (FSL) preserves the parallel model training principle of FL, with a reduced device computation requirement thanks to splitting the ML model between the server and clients. However, FSL still incurs very high communication overhead due to transmitting the smashed data and gradients between the clients and the server in each global round. Furthermore, the server has to maintain separate models for every client, resulting in a significant computation and storage requirement that grows linearly with the number of clients. This paper tries to solve these two issues by proposing a communication and storage efficient federated and split learning (CSE-FSL) strategy, which utilizes an auxiliary network to locally update the client models while keeping only a single model at the server, hence avoiding the communication of gradients from the server and greatly reducing the server resource requirement. Communication cost is further reduced by only sending the smashed data in selected epochs from the clients. We provide a rigorous theoretical analysis of CSE-FSL that guarantees its convergence for non-convex loss functions. Extensive experimental results demonstrate that CSE-FSL has a significant communication reduction over existing FSL techniques while achieving state-of-the-art convergence and model accuracy, using several real-world FL tasks.

Related papers

• Hyperdimensional Computing Empowered Federated Foundation Model over Wireless Networks for Metaverse [56.384390765357004]
  We propose an integrated federated split learning and hyperdimensional computing framework for emerging foundation models. This novel approach reduces communication costs, computation load, and privacy risks, making it suitable for resource-constrained edge devices in the Metaverse.
  arXiv  Detail & Related papers  (2024-08-26T17:03:14Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach [67.27031215756121]
  Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as emphConfederated Learning (CFL) in order to accommodate a larger number of users.
  arXiv  Detail & Related papers  (2024-02-28T03:27:10Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach [16.093068118849246]
  This paper presents a novel split federated learning (SFL) framework that pairs clients with different computational resources. A greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem. Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance.
  arXiv  Detail & Related papers  (2023-08-26T11:10:54Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• PFSL: Personalized & Fair Split Learning with Data & Label Privacy for thin clients [0.5144809478361603]
  PFSL is a new framework of distributed split learning where a large number of thin clients perform transfer learning in parallel. We implement a lightweight step of personalization of client models to provide high performance for their respective data distributions. Our accuracy far exceeds that of current algorithms SL and is very close to that of centralized learning on several real-life benchmarks.
  arXiv  Detail & Related papers  (2023-03-19T10:38:29Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• Over-The-Air Clustered Wireless Federated Learning [2.2530496464901106]
  Over-the-air (OTA) FL is preferred since the clients can transmit parameter updates simultaneously to a server. In the absence of a powerful server, decentralised strategy is employed where clients communicate with their neighbors to obtain a consensus ML model. We propose the OTA semi-decentralised clustered wireless FL (CWFL) and CWFL-Prox algorithms, which is communication efficient as compared to the decentralised FL strategy.
  arXiv  Detail & Related papers  (2022-11-07T08:34:35Z)
• Communication-Efficient Online Federated Learning Framework for Nonlinear Regression [5.67468104295976]
  This paper presents a partial-sharing-based online federated learning framework (PSO-Fed) PSO-Fed enables clients to update their local models using continuous streaming data and share only portions of those updated models with the server. Experimental results show that PSO-Fed can achieve competitive performance with a significantly lower communication overhead than Online-Fed.
  arXiv  Detail & Related papers  (2021-10-13T08:11:34Z)
• Coded Computing for Federated Learning at the Edge [3.385874614913973]
  Federated Learning (FL) enables training a global model from data generated locally at the client nodes, without moving client data to a centralized server. Recent work proposes to mitigate stragglers and speed up training for linear regression tasks by assigning redundant computations at the MEC server. We develop CodedFedL that addresses the difficult task of extending CFL to distributed non-linear regression and classification problems with multioutput labels.
  arXiv  Detail & Related papers  (2020-07-07T08:20:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.