CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated
Learning
- URL: http://arxiv.org/abs/2210.02912v1
- Date: Thu, 6 Oct 2022 13:30:16 GMT
- Title: CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated
Learning
- Authors: Samuel Maddock, Alexandre Sablayrolles and Pierre Stock
- Abstract summary: Federated Learning (FL) is a setting for training machine learning models in distributed environments.
We propose a novel method, CANIFE, that uses carefully crafted samples by a strong adversary to evaluate the empirical privacy of a training round.
- Score: 77.27443885999404
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning (FL) is a setting for training machine learning models in
distributed environments where the clients do not share their raw data but
instead send model updates to a server. However, model updates can be subject
to attacks and leak private information. Differential Privacy (DP) is a leading
mitigation strategy which involves adding noise to clipped model updates,
trading off performance for strong theoretical privacy guarantees. Previous
work has shown that the threat model of DP is conservative and that the
obtained guarantees may be vacuous or may not directly translate to information
leakage in practice. In this paper, we aim to achieve a tighter measurement of
the model exposure by considering a realistic threat model. We propose a novel
method, CANIFE, that uses canaries - carefully crafted samples by a strong
adversary to evaluate the empirical privacy of a training round. We apply this
attack to vision models trained on CIFAR-10 and CelebA and to language models
trained on Sent140 and Shakespeare. In particular, in realistic FL scenarios,
we demonstrate that the empirical epsilon obtained with CANIFE is 2-7x lower
than the theoretical bound.
Related papers
- FullCert: Deterministic End-to-End Certification for Training and Inference of Neural Networks [62.897993591443594]
FullCert is the first end-to-end certifier with sound, deterministic bounds, which proves robustness against both training-time and inference-time attacks.
We combine our theoretical work with a new open-source library BoundFlow, which enables model training on bounded datasets.
arXiv Detail & Related papers (2024-06-17T13:23:52Z) - Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
In this paper, we unveil a new vulnerability: the privacy backdoor attack.
When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model.
Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
arXiv Detail & Related papers (2024-04-01T16:50:54Z) - Assessing Privacy Risks in Language Models: A Case Study on
Summarization Tasks [65.21536453075275]
We focus on the summarization task and investigate the membership inference (MI) attack.
We exploit text similarity and the model's resistance to document modifications as potential MI signals.
We discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.
arXiv Detail & Related papers (2023-10-20T05:44:39Z) - TrustGAN: Training safe and trustworthy deep learning models through
generative adversarial networks [0.0]
We present TrustGAN, a generative adversarial network pipeline targeting trustness.
The pipeline can accept any given deep learning model which outputs a prediction and a confidence on this prediction.
It is applied here to a target classification model trained on MNIST data to recognise numbers based on images.
arXiv Detail & Related papers (2022-11-25T09:57:23Z) - Careful What You Wish For: on the Extraction of Adversarially Trained
Models [2.707154152696381]
Recent attacks on Machine Learning (ML) models pose several security and privacy threats.
We propose a framework to assess extraction attacks on adversarially trained models.
We show that adversarially trained models are more vulnerable to extraction attacks than models obtained under natural training circumstances.
arXiv Detail & Related papers (2022-07-21T16:04:37Z) - Do Gradient Inversion Attacks Make Federated Learning Unsafe? [70.0231254112197]
Federated learning (FL) allows the collaborative training of AI models without needing to share raw data.
Recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data.
In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack.
arXiv Detail & Related papers (2022-02-14T18:33:12Z) - Knowledge-Enriched Distributional Model Inversion Attacks [49.43828150561947]
Model inversion (MI) attacks are aimed at reconstructing training data from model parameters.
We present a novel inversion-specific GAN that can better distill knowledge useful for performing attacks on private models from public data.
Our experiments show that the combination of these techniques can significantly boost the success rate of the state-of-the-art MI attacks by 150%.
arXiv Detail & Related papers (2020-10-08T16:20:48Z) - Privacy Analysis of Deep Learning in the Wild: Membership Inference
Attacks against Transfer Learning [27.494206948563885]
We present the first systematic evaluation of membership inference attacks against transfer learning models.
Experiments on four real-world image datasets show that membership inference can achieve effective performance.
Our results shed light on the severity of membership risks stemming from machine learning models in practice.
arXiv Detail & Related papers (2020-09-10T14:14:22Z) - Trade-offs between membership privacy & adversarially robust learning [13.37805637358556]
We identify settings where standard models will overfit to a larger extent in comparison to robust models.
The degree of overfitting naturally depends on the amount of data available for training.
arXiv Detail & Related papers (2020-06-08T14:20:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.