Preserving Privacy in Federated Learning with Ensemble Cross-Domain
Knowledge Distillation
- URL: http://arxiv.org/abs/2209.04599v1
- Date: Sat, 10 Sep 2022 05:20:31 GMT
- Title: Preserving Privacy in Federated Learning with Ensemble Cross-Domain
Knowledge Distillation
- Authors: Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence
Chen, David Doermann, Arun Innanje
- Abstract summary: Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model.
Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution.
We develop a privacy preserving and communication efficient method in a FL framework with one-shot offline knowledge distillation.
- Score: 22.151404603413752
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning (FL) is a machine learning paradigm where local nodes
collaboratively train a central model while the training data remains
decentralized. Existing FL methods typically share model parameters or employ
co-distillation to address the issue of unbalanced data distribution. However,
they suffer from communication bottlenecks. More importantly, they risk privacy
leakage. In this work, we develop a privacy preserving and communication
efficient method in a FL framework with one-shot offline knowledge distillation
using unlabeled, cross-domain public data. We propose a quantized and noisy
ensemble of local predictions from completely trained local models for stronger
privacy guarantees without sacrificing accuracy. Based on extensive experiments
on image classification and text classification tasks, we show that our
privacy-preserving method outperforms baseline FL algorithms with superior
performance in both accuracy and communication efficiency.
Related papers
- Federated Learning via Input-Output Collaborative Distillation [40.38454921071808]
Federated learning (FL) is a machine learning paradigm in which distributed local nodes collaboratively train a central model without sharing individually held private data.
We propose a data-free FL framework based on local-to-central collaborative distillation with direct input and output space exploitation.
arXiv Detail & Related papers (2023-12-22T07:05:13Z) - Federated Learning with Reduced Information Leakage and Computation [20.005520306964485]
Federated learning (FL) is a distributed learning paradigm that allows multiple decentralized clients to collaboratively learn a common model without sharing local data.
In this paper, we introduce Upcycled-FL, a novel federated learning framework with first-order approximation applied at every even iteration.
Under this framework, half of the FL updates incur no information leakage and require much less computation.
arXiv Detail & Related papers (2023-10-10T06:22:06Z) - Benchmarking FedAvg and FedCurv for Image Classification Tasks [1.376408511310322]
This paper focuses on the problem of statistical heterogeneity of the data in the same federated network.
Several Federated Learning algorithms, such as FedAvg, FedProx and Federated Curvature (FedCurv) have already been proposed.
As a side product of this work, we release the non-IID version of the datasets we used so to facilitate further comparisons from the FL community.
arXiv Detail & Related papers (2023-03-31T10:13:01Z) - Federated Learning with Privacy-Preserving Ensemble Attention
Distillation [63.39442596910485]
Federated Learning (FL) is a machine learning paradigm where many local nodes collaboratively train a central model while keeping the training data decentralized.
We propose a privacy-preserving FL framework leveraging unlabeled public data for one-way offline knowledge distillation.
Our technique uses decentralized and heterogeneous local data like existing FL approaches, but more importantly, it significantly reduces the risk of privacy leakage.
arXiv Detail & Related papers (2022-10-16T06:44:46Z) - Acceleration of Federated Learning with Alleviated Forgetting in Local
Training [61.231021417674235]
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy.
We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage.
Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
arXiv Detail & Related papers (2022-03-05T02:31:32Z) - Personalization Improves Privacy-Accuracy Tradeoffs in Federated
Optimization [57.98426940386627]
We show that coordinating local learning with private centralized learning yields a generically useful and improved tradeoff between accuracy and privacy.
We illustrate our theoretical results with experiments on synthetic and real-world datasets.
arXiv Detail & Related papers (2022-02-10T20:44:44Z) - Local Learning Matters: Rethinking Data Heterogeneity in Federated
Learning [61.488646649045215]
Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
arXiv Detail & Related papers (2021-11-28T19:03:39Z) - FedMix: Approximation of Mixup under Mean Augmented Federated Learning [60.503258658382]
Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device.
Current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases.
We propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup.
arXiv Detail & Related papers (2021-07-01T06:14:51Z) - FedOCR: Communication-Efficient Federated Learning for Scene Text
Recognition [76.26472513160425]
We study how to make use of decentralized datasets for training a robust scene text recognizer.
To make FedOCR fairly suitable to be deployed on end devices, we make two improvements including using lightweight models and hashing techniques.
arXiv Detail & Related papers (2020-07-22T14:30:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.