FedSiKD: Clients Similarity and Knowledge Distillation: Addressing
Non-i.i.d. and Constraints in Federated Learning
- URL: http://arxiv.org/abs/2402.09095v1
- Date: Wed, 14 Feb 2024 11:16:50 GMT
- Title: FedSiKD: Clients Similarity and Knowledge Distillation: Addressing
Non-i.i.d. and Constraints in Federated Learning
- Authors: Yousef Alsenani, Rahul Mishra, Khaled R. Ahmed, Atta Ur Rahman
- Abstract summary: We introduce FedSiKD, which incorporates knowledge distillation (KD) within a similarity-based federated learning framework.
As clients join the system, they securely share relevant statistics about their data distribution, promoting intra-cluster homogeneity.
FedSiKD outperforms state-of-the-art algorithms by exceeding by 25% and 18% for highly skewed data at $alpha = 0.1,0.5$ on the HAR and MNIST datasets.
- Score: 7.718401895021425
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: In recent years, federated learning (FL) has emerged as a promising technique
for training machine learning models in a decentralized manner while also
preserving data privacy. The non-independent and identically distributed
(non-i.i.d.) nature of client data, coupled with constraints on client or edge
devices, presents significant challenges in FL. Furthermore, learning across a
high number of communication rounds can be risky and potentially unsafe for
model exploitation. Traditional FL approaches may suffer from these challenges.
Therefore, we introduce FedSiKD, which incorporates knowledge distillation (KD)
within a similarity-based federated learning framework. As clients join the
system, they securely share relevant statistics about their data distribution,
promoting intra-cluster homogeneity. This enhances optimization efficiency and
accelerates the learning process, effectively transferring knowledge between
teacher and student models and addressing device constraints. FedSiKD
outperforms state-of-the-art algorithms by achieving higher accuracy, exceeding
by 25\% and 18\% for highly skewed data at $\alpha = {0.1,0.5}$ on the HAR and
MNIST datasets, respectively. Its faster convergence is illustrated by a 17\%
and 20\% increase in accuracy within the first five rounds on the HAR and MNIST
datasets, respectively, highlighting its early-stage learning proficiency. Code
is publicly available and hosted on GitHub (https://github.com/SimuEnv/FedSiKD)
Related papers
- Fed-Credit: Robust Federated Learning with Credibility Management [18.349127735378048]
Federated Learning (FL) is an emerging machine learning approach enabling model training on decentralized devices or data sources.
We propose a robust FL approach based on the credibility management scheme, called Fed-Credit.
The results exhibit superior accuracy and resilience against adversarial attacks, all while maintaining comparatively low computational complexity.
arXiv Detail & Related papers (2024-05-20T03:35:13Z) - An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Benchmarking FedAvg and FedCurv for Image Classification Tasks [1.376408511310322]
This paper focuses on the problem of statistical heterogeneity of the data in the same federated network.
Several Federated Learning algorithms, such as FedAvg, FedProx and Federated Curvature (FedCurv) have already been proposed.
As a side product of this work, we release the non-IID version of the datasets we used so to facilitate further comparisons from the FL community.
arXiv Detail & Related papers (2023-03-31T10:13:01Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - Preserving Privacy in Federated Learning with Ensemble Cross-Domain
Knowledge Distillation [22.151404603413752]
Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model.
Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution.
We develop a privacy preserving and communication efficient method in a FL framework with one-shot offline knowledge distillation.
arXiv Detail & Related papers (2022-09-10T05:20:31Z) - Acceleration of Federated Learning with Alleviated Forgetting in Local
Training [61.231021417674235]
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy.
We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage.
Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
arXiv Detail & Related papers (2022-03-05T02:31:32Z) - Stochastic Coded Federated Learning with Convergence and Privacy
Guarantees [8.2189389638822]
Federated learning (FL) has attracted much attention as a privacy-preserving distributed machine learning framework.
This paper proposes a coded federated learning framework, namely coded federated learning (SCFL) to mitigate the straggler issue.
We characterize the privacy guarantee by the mutual information differential privacy (MI-DP) and analyze the convergence performance in federated learning.
arXiv Detail & Related papers (2022-01-25T04:43:29Z) - Local Learning Matters: Rethinking Data Heterogeneity in Federated
Learning [61.488646649045215]
Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
arXiv Detail & Related papers (2021-11-28T19:03:39Z) - CatFedAvg: Optimising Communication-efficiency and Classification
Accuracy in Federated Learning [2.2172881631608456]
We introduce a new family of Federated Learning algorithms called CatFedAvg.
It improves the communication efficiency but improves the quality of learning using a category coverage inNIST strategy.
Our experiments show that an increase of 10% absolute points accuracy using the M dataset with 70% absolute points lower network transfer over FedAvg.
arXiv Detail & Related papers (2020-11-14T06:52:02Z) - WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices.
We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
arXiv Detail & Related papers (2020-08-13T04:26:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.