Convergence Analysis of Split Federated Learning on Heterogeneous Data

• URL: http://arxiv.org/abs/2402.15166v3
• Date: Thu, 09 Jan 2025 11:39:19 GMT
• Title: Convergence Analysis of Split Federated Learning on Heterogeneous Data
• Authors: Pengchao Han, Chao Huang, Geng Tian, Ming Tang, Xin Liu,
• Abstract summary: Split learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and the other trains the other. We provide convergence analysis of SFL for strongly convex and general objectives on heterogeneous data.
• Score: 10.61370409320618
• License:
• Abstract: Split federated learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL), due to the potential dual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data. The convergence rates are $O(1/T)$ and $O(1/\sqrt[3]{T})$, respectively, where $T$ denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and the scenario where some clients may be unavailable during training. Experimental experiments validate our theoretical results and show that SFL outperforms FL and split learning (SL) when data is highly heterogeneous across a large number of clients.

Related papers

• The Impact of Cut Layer Selection in Split Federated Learning [6.481423646861632]
  Split Federated Learning (SFL) is a distributed machine learning paradigm that combines federated learning and split learning. In SFL, a neural network is partitioned at a cut layer, with the initial layers deployed on clients and remaining layers on a training server.
  arXiv  Detail & Related papers  (2024-12-20T03:52:54Z)
• SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning [12.542161138042632]
  Decentralized federated learning (DFL) realizes cooperative model training among connected clients without relying on a central server. Most existing work on DFL focuses on supervised learning, assuming each client possesses sufficient labeled data for local training. We propose SemiDFL, the first semi-supervised DFL method that enhances DFL performance in SSL scenarios by establishing a consensus in both data and model spaces.
  arXiv  Detail & Related papers  (2024-12-18T08:12:55Z)
• How Can Incentives and Cut Layer Selection Influence Data Contribution in Split Federated Learning? [49.16923922018379]
  Split Federated Learning (SFL) has emerged as a promising approach by combining the advantages of federated and split learning. We model the problem using a hierarchical decision-making approach, formulated as a single-leader multi-follower Stackelberg game. Our findings show that the Stackelberg equilibrium solution maximizes the utility for both the clients and the SFL model owner.
  arXiv  Detail & Related papers  (2024-12-10T06:24:08Z)
• FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion [48.90879664138855]
  One-shot Federated Learning (OFL) significantly reduces communication costs in FL by aggregating trained models only once. However, the performance of advanced OFL methods is far behind the normal FL. We propose a novel learning approach to endow OFL with superb performance and low communication and storage costs, termed as FuseFL.
  arXiv  Detail & Related papers  (2024-10-27T09:07:10Z)
• Convergence Analysis of Sequential Federated Learning on Heterogeneous Data [5.872735527071425]
  There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) FL (SFL) where clients train in a sequential manner. In this paper, we establish the convergence guarantees SFL on heterogeneous data is still lacking. Experimental results validate the counterintuitive analysis result that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.
  arXiv  Detail & Related papers  (2023-11-06T14:48:51Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• PFL-GAN: When Client Heterogeneity Meets Generative Models in Personalized Federated Learning [55.930403371398114]
  We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL) PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation. The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
  arXiv  Detail & Related papers  (2023-08-23T22:38:35Z)
• SplitFed resilience to packet loss: Where to split, that is the question [27.29876880765472]
  Split Federated Learning (SFL) aims to reduce the computational power required by each client in FL and parallelize SL while maintaining privacy. This paper investigates the robustness of SFL against packet loss on communication links. Experiments are carried out on a segmentation model for human embryo images and indicate the statistically significant advantage of a deeper split point.
  arXiv  Detail & Related papers  (2023-07-25T22:54:47Z)
• Improving the Model Consistency of Decentralized Federated Learning [68.2795379609854]
  Federated Learning (FL) discards the central server and each client only communicates with its neighbors in a decentralized communication network. Existing DFL suffers from inconsistency among local clients, which results in inferior compared to FLFL. We propose DFedSAMMGS, where $1lambda$ is the spectral gossip matrix and $Q$ is the number of sparse data gaps.
  arXiv  Detail & Related papers  (2023-02-08T14:37:34Z)
• Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
  Federated learning (FL) is vulnerable to heterogeneously distributed data. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
  arXiv  Detail & Related papers  (2022-01-27T08:43:11Z)
• Splitfed learning without client-side synchronization: Analyzing client-side split network portion size to overall performance [4.689140226545214]
  Federated Learning (FL), Split Learning (SL), and SplitFed Learning (SFL) are three recent developments in distributed machine learning. This paper studies SFL without client-side model synchronization. It provides only 1%-2% better accuracy than Multi-head Split Learning on the MNIST test set.
  arXiv  Detail & Related papers  (2021-09-19T22:57:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.