SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments

• URL: http://arxiv.org/abs/2405.04875v1
• Date: Wed, 8 May 2024 08:12:21 GMT
• Title: SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments
• Authors: Jiarong Yang, Yuan Liu,
• Abstract summary: Split Federated Learning (SFL) is a distributed machine learning framework which strategically divides the learning process between a server and clients. We propose SFL with Concatenated Activations and Logit Adjustments (SCALA) to deal with the label distribution variation across different subsets of participating clients.
• Score: 4.826527560777905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Split Federated Learning (SFL) is a distributed machine learning framework which strategically divides the learning process between a server and clients and collaboratively trains a shared model by aggregating local models updated based on data from distributed clients. However, data heterogeneity and partial client participation result in label distribution skew, which severely degrades the learning performance. To address this issue, we propose SFL with Concatenated Activations and Logit Adjustments (SCALA). Specifically, the activations from the client-side models are concatenated as the input of the server-side model so as to centrally adjust label distribution across different clients, and logit adjustments of loss functions on both server-side and client-side models are performed to deal with the label distribution variation across different subsets of participating clients. Theoretical analysis and experimental results verify the superiority of the proposed SCALA on public datasets.

Related papers

• A Bayesian Framework for Clustered Federated Learning [14.426129993432193]
  One of the main challenges of federated learning (FL) is handling non-independent and identically distributed (non-IID) client data. We present a unified Bayesian framework for clustered FL which associates clients to clusters. This work provides insights on client-cluster associations and enables client knowledge sharing in new ways.
  arXiv  Detail & Related papers  (2024-10-20T19:11:24Z)
• FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering [26.478852701376294]
  Federated learning (FL) is an emerging distributed machine learning paradigm. One of the major challenges in FL is the presence of uneven data distributions across client devices. We propose em FedClust, a novel approach for CFL that leverages the correlation between local model weights and the data distribution of clients.
  arXiv  Detail & Related papers  (2024-07-09T02:47:16Z)
• Towards Client Driven Federated Learning [7.528642177161784]
  We introduce Client-Driven Federated Learning (CDFL), a novel FL framework that puts clients at the driving role. In CDFL, each client independently and asynchronously updates its model by uploading the locally trained model to the server and receiving a customized model tailored to its local task.
  arXiv  Detail & Related papers  (2024-05-24T10:17:49Z)
• FedClust: Optimizing Federated Learning on Non-IID Data through Weight-Driven Client Clustering [28.057411252785176]
  Federated learning (FL) is an emerging distributed machine learning paradigm enabling collaborative model training on decentralized devices without exposing their local data. This paper proposes FedClust, a novel CFL approach leveraging correlations between local model weights and client data distributions.
  arXiv  Detail & Related papers  (2024-03-07T01:50:36Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction [76.26710990597498]
  We show that the class-imbalance of the grouped data from randomly selected clients can lead to significant performance degradation. Based on our key observation, we design an efficient client sampling mechanism, i.e., Federated Class-balanced Sampling (Fed-CBS) In particular, we propose a measure of class-imbalance and then employ homomorphic encryption to derive this measure in a privacy-preserving way.
  arXiv  Detail & Related papers  (2022-09-30T05:42:56Z)
• Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces [59.33965805898736]
  Clustered learning has been shown to produce promising results by grouping clients into clusters. Existing FL algorithms are essentially trying to group clients together with similar distributions. Prior FL algorithms attempt similarities indirectly during training.
  arXiv  Detail & Related papers  (2022-09-21T17:37:54Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)
• Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocation [119.19061102064497]
  We propose a decentralized FL framework by integrating blockchain into FL, namely, blockchain assisted decentralized federated learning (BLADE-FL) In a round of the proposed BLADE-FL, each client broadcasts its trained model to other clients, competes to generate a block based on the received models, and then aggregates the models from the generated block before its local training of the next round. We explore the impact of lazy clients on the learning performance of BLADE-FL, and characterize the relationship among the optimal K, the learning parameters, and the proportion of lazy clients.
  arXiv  Detail & Related papers  (2021-01-18T07:19:08Z)
• Client Adaptation improves Federated Learning with Simulated Non-IID Clients [1.0896567381206714]
  We present a federated learning approach for learning a client adaptable, robust model when data is non-identically and non-independently distributed (non-IID) across clients. We show that adding learned client-specific conditioning improves model performance, and the approach is shown to work on balanced and imbalanced data set from both audio and image domains.
  arXiv  Detail & Related papers  (2020-07-09T13:48:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.