Differentially Private Federated Learning with Laplacian Smoothing
- URL: http://arxiv.org/abs/2005.00218v2
- Date: Fri, 10 Sep 2021 05:36:11 GMT
- Title: Differentially Private Federated Learning with Laplacian Smoothing
- Authors: Zhicong Liang, Bao Wang, Quanquan Gu, Stanley Osher, Yuan Yao
- Abstract summary: Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users.
An adversary may still be able to infer the private training data by attacking the released model.
Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
- Score: 72.85272874099644
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning aims to protect data privacy by collaboratively learning a
model without sharing private data among users. However, an adversary may still
be able to infer the private training data by attacking the released model.
Differential privacy provides a statistical protection against such attacks at
the price of significantly degrading the accuracy or utility of the trained
models. In this paper, we investigate a utility enhancement scheme based on
Laplacian smoothing for differentially private federated learning (DP-Fed-LS),
where the parameter aggregation with injected Gaussian noise is improved in
statistical precision without losing privacy budget. Our key observation is
that the aggregated gradients in federated learning often enjoy a type of
smoothness, i.e. sparsity in the graph Fourier basis with polynomial decays of
Fourier coefficients as frequency grows, which can be exploited by the
Laplacian smoothing efficiently. Under a prescribed differential privacy
budget, convergence error bounds with tight rates are provided for DP-Fed-LS
with uniform subsampling of heterogeneous Non-IID data, revealing possible
utility improvement of Laplacian smoothing in effective dimensionality and
variance reduction, among others. Experiments over MNIST, SVHN, and Shakespeare
datasets show that the proposed method can improve model accuracy with
DP-guarantee and membership privacy under both uniform and Poisson subsampling
mechanisms.
Related papers
- Mitigating Disparate Impact of Differential Privacy in Federated Learning through Robust Clustering [4.768272342753616]
Federated Learning (FL) is a decentralized machine learning (ML) approach that keeps data localized and often incorporates Differential Privacy (DP) to enhance privacy guarantees.
Recent work has attempted to address performance fairness in vanilla FL through clustering, but this method remains sensitive and prone to errors.
We propose a novel clustered DPFL algorithm designed to effectively identify clients' clusters in highly heterogeneous settings.
arXiv Detail & Related papers (2024-05-29T17:03:31Z) - Privacy Amplification for the Gaussian Mechanism via Bounded Support [64.86780616066575]
Data-dependent privacy accounting frameworks such as per-instance differential privacy (pDP) and Fisher information loss (FIL) confer fine-grained privacy guarantees for individuals in a fixed training dataset.
We propose simple modifications of the Gaussian mechanism with bounded support, showing that they amplify privacy guarantees under data-dependent accounting.
arXiv Detail & Related papers (2024-03-07T21:22:07Z) - Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients.
FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification.
Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
arXiv Detail & Related papers (2023-05-01T20:04:46Z) - Over-the-Air Federated Learning with Privacy Protection via Correlated
Additive Perturbations [57.20885629270732]
We consider privacy aspects of wireless federated learning with Over-the-Air (OtA) transmission of gradient updates from multiple users/agents to an edge server.
Traditional perturbation-based methods provide privacy protection while sacrificing the training accuracy.
In this work, we aim at minimizing privacy leakage to the adversary and the degradation of model accuracy at the edge server.
arXiv Detail & Related papers (2022-10-05T13:13:35Z) - Large Scale Transfer Learning for Differentially Private Image
Classification [51.10365553035979]
Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy.
Private training using DP-SGD protects against leakage by injecting noise into individual example gradients.
While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
arXiv Detail & Related papers (2022-05-06T01:22:20Z) - Federated Learning with Sparsified Model Perturbation: Improving
Accuracy under Client-Level Differential Privacy [27.243322019117144]
Federated learning (FL) enables distributed clients to collaboratively learn a shared statistical model.
sensitive information about the training data can still be inferred from model updates shared in FL.
Differential privacy (DP) is the state-of-the-art technique to defend against those attacks.
This paper develops a novel FL scheme named Fed-SMP that provides client-level DP guarantee while maintaining high model accuracy.
arXiv Detail & Related papers (2022-02-15T04:05:42Z) - Stochastic Coded Federated Learning with Convergence and Privacy
Guarantees [8.2189389638822]
Federated learning (FL) has attracted much attention as a privacy-preserving distributed machine learning framework.
This paper proposes a coded federated learning framework, namely coded federated learning (SCFL) to mitigate the straggler issue.
We characterize the privacy guarantee by the mutual information differential privacy (MI-DP) and analyze the convergence performance in federated learning.
arXiv Detail & Related papers (2022-01-25T04:43:29Z) - Don't Generate Me: Training Differentially Private Generative Models
with Sinkhorn Divergence [73.14373832423156]
We propose DP-Sinkhorn, a novel optimal transport-based generative method for learning data distributions from private data with differential privacy.
Unlike existing approaches for training differentially private generative models, we do not rely on adversarial objectives.
arXiv Detail & Related papers (2021-11-01T18:10:21Z) - Federated Model Distillation with Noise-Free Differential Privacy [35.72801867380072]
We propose a novel framework called FEDMD-NFDP, which applies a Noise-Free Differential Privacy (NFDP) mechanism into a federated model distillation framework.
Our extensive experimental results on various datasets validate that FEDMD-NFDP can deliver comparable utility and communication efficiency.
arXiv Detail & Related papers (2020-09-11T17:19:56Z) - Federated Learning with Sparsification-Amplified Privacy and Adaptive
Optimization [27.243322019117144]
Federated learning (FL) enables distributed agents to collaboratively learn a centralized model without sharing their raw data with each other.
We propose a new FL framework with sparsification-amplified privacy.
Our approach integrates random sparsification with gradient perturbation on each agent to amplify privacy guarantee.
arXiv Detail & Related papers (2020-08-01T20:22:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.