Differentially Private Decentralized Deep Learning with Consensus
Algorithms
- URL: http://arxiv.org/abs/2306.13892v1
- Date: Sat, 24 Jun 2023 07:46:00 GMT
- Title: Differentially Private Decentralized Deep Learning with Consensus
Algorithms
- Authors: Jasmine Bayrooti, Zhan Gao, Amanda Prorok
- Abstract summary: Cooperative decentralized deep learning relies on direct information exchange between communicating agents.
Sharing parameters with untrustworthy neighboring agents could leak exploitable information about local datasets.
We introduce differentially private decentralized learning that secures each agent's local dataset during and after cooperative training.
- Score: 10.208363125551555
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Cooperative decentralized deep learning relies on direct information exchange
between communicating agents, each with access to a local dataset which should
be kept private. The goal is for all agents to achieve consensus on model
parameters after training. However, sharing parameters with untrustworthy
neighboring agents could leak exploitable information about local datasets. To
combat this, we introduce differentially private decentralized learning that
secures each agent's local dataset during and after cooperative training. In
our approach, we generalize Differentially Private Stochastic Gradient Descent
(DP-SGD) -- a popular differentially private training method for centralized
deep learning -- to practical subgradient- and ADMM-based decentralized
learning methods. Our algorithms' differential privacy guarantee holds for
arbitrary deep learning objective functions, and we analyze the convergence
properties for strongly convex objective functions. We compare our algorithms
against centrally trained models on standard classification tasks and evaluate
the relationships between performance, privacy budget, graph connectivity, and
degree of training data overlap among agents. We find that differentially
private gradient tracking is resistant to performance degradation under sparse
graphs and non-uniform data distributions. Furthermore, we show that it is
possible to learn a model achieving high accuracies, within 3% of DP-SGD on
MNIST under (1, 10^-5)-differential privacy and within 6% of DP-SGD on
CIFAR-100 under (10, 10^-5)-differential privacy, without ever sharing raw data
with other agents. Open source code can be found at:
https://github.com/jbayrooti/dp-dec-learning.
Related papers
- Locally Differentially Private Gradient Tracking for Distributed Online
Learning over Directed Graphs [2.1271873498506038]
We propose a locally differentially private gradient tracking based distributed online learning algorithm.
We prove that the proposed algorithm converges in mean square to the exact optimal solution while ensuring rigorous local differential privacy.
arXiv Detail & Related papers (2023-10-24T18:15:25Z) - Fine-Tuning with Differential Privacy Necessitates an Additional
Hyperparameter Search [38.83524780461911]
We show how carefully selecting the layers being fine-tuned in the pretrained neural network allows us to establish new state-of-the-art tradeoffs between privacy and accuracy.
We achieve 77.9% accuracy for $(varepsilon, delta)= (2, 10-5)$ on CIFAR-100 for a model pretrained on ImageNet.
arXiv Detail & Related papers (2022-10-05T11:32:49Z) - Preserving Privacy in Federated Learning with Ensemble Cross-Domain
Knowledge Distillation [22.151404603413752]
Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model.
Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution.
We develop a privacy preserving and communication efficient method in a FL framework with one-shot offline knowledge distillation.
arXiv Detail & Related papers (2022-09-10T05:20:31Z) - Differentially Private Vertical Federated Clustering [13.27934054846057]
In many applications, multiple parties have private data regarding the same set of users but on disjoint sets of attributes.
To enable model learning while protecting the privacy of the data subjects, we need vertical federated learning (VFL) techniques.
The algorithm proposed in this paper is the first practical solution for differentially private vertical federated k-means clustering.
arXiv Detail & Related papers (2022-08-02T19:23:48Z) - Mixed Differential Privacy in Computer Vision [133.68363478737058]
AdaMix is an adaptive differentially private algorithm for training deep neural network classifiers using both private and public image data.
A few-shot or even zero-shot learning baseline that ignores private data can outperform fine-tuning on a large private dataset.
arXiv Detail & Related papers (2022-03-22T06:15:43Z) - Personalization Improves Privacy-Accuracy Tradeoffs in Federated
Optimization [57.98426940386627]
We show that coordinating local learning with private centralized learning yields a generically useful and improved tradeoff between accuracy and privacy.
We illustrate our theoretical results with experiments on synthetic and real-world datasets.
arXiv Detail & Related papers (2022-02-10T20:44:44Z) - Weight Divergence Driven Divide-and-Conquer Approach for Optimal
Federated Learning from non-IID Data [0.0]
Federated Learning allows training of data stored in distributed devices without the need for centralizing training data.
We propose a novel Divide-and-Conquer training methodology that enables the use of the popular FedAvg aggregation algorithm.
arXiv Detail & Related papers (2021-06-28T09:34:20Z) - Quasi-Global Momentum: Accelerating Decentralized Deep Learning on
Heterogeneous Data [77.88594632644347]
Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks.
In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge.
We propose a novel momentum-based method to mitigate this decentralized training difficulty.
arXiv Detail & Related papers (2021-02-09T11:27:14Z) - Graph-Homomorphic Perturbations for Private Decentralized Learning [64.26238893241322]
Local exchange of estimates allows inference of data based on private data.
perturbations chosen independently at every agent, resulting in a significant performance loss.
We propose an alternative scheme, which constructs perturbations according to a particular nullspace condition, allowing them to be invisible.
arXiv Detail & Related papers (2020-10-23T10:35:35Z) - Decentralised Learning from Independent Multi-Domain Labels for Person
Re-Identification [69.29602103582782]
Deep learning has been successful for many computer vision tasks due to the availability of shared and centralised large-scale training data.
However, increasing awareness of privacy concerns poses new challenges to deep learning, especially for person re-identification (Re-ID)
We propose a novel paradigm called Federated Person Re-Identification (FedReID) to construct a generalisable global model (a central server) by simultaneously learning with multiple privacy-preserved local models (local clients)
This client-server collaborative learning process is iteratively performed under privacy control, enabling FedReID to realise decentralised learning without sharing distributed data nor collecting any
arXiv Detail & Related papers (2020-06-07T13:32:33Z) - User-Level Privacy-Preserving Federated Learning: Analysis and
Performance Optimization [77.43075255745389]
Federated learning (FL) is capable of preserving private data from mobile terminals (MTs) while training the data into useful models.
From a viewpoint of information theory, it is still possible for a curious server to infer private information from the shared models uploaded by MTs.
We propose a user-level differential privacy (UDP) algorithm by adding artificial noise to the shared models before uploading them to servers.
arXiv Detail & Related papers (2020-02-29T10:13:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.