Measuring Unintended Memorisation of Unique Private Features in Neural Networks

• URL: http://arxiv.org/abs/2202.08099v1
• Date: Wed, 16 Feb 2022 14:39:05 GMT
• Title: Measuring Unintended Memorisation of Unique Private Features in Neural Networks
• Authors: John Hartley, Sotirios A. Tsaftaris
• Abstract summary: We show that neural networks unintentionally memorise unique features even when they occur only once in training data. An example of a unique feature is a person's name that is accidentally present on a training image.
• Score: 15.174895411434026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks pose a privacy risk to training data due to their propensity to memorise and leak information. Focusing on image classification, we show that neural networks also unintentionally memorise unique features even when they occur only once in training data. An example of a unique feature is a person's name that is accidentally present on a training image. Assuming access to the inputs and outputs of a trained model, the domain of the training data, and knowledge of unique features, we develop a score estimating the model's sensitivity to a unique feature by comparing the KL divergences of the model's output distributions given modified out-of-distribution images. Our results suggest that unique features are memorised by multi-layer perceptrons and convolutional neural networks trained on benchmark datasets, such as MNIST, Fashion-MNIST and CIFAR-10. We find that strategies to prevent overfitting (e.g.\ early stopping, regularisation, batch normalisation) do not prevent memorisation of unique features. These results imply that neural networks pose a privacy risk to rarely occurring private information. These risks can be more pronounced in healthcare applications if patient information is present in the training data.

Related papers

• Accurate Forgetting for All-in-One Image Restoration Model [3.367455972998532]
  Currently, a low-cost scheme called Machine Unlearning forgets the private data remembered in the model. Inspired by this, we try to use this concept to bridge the gap between the fields of image restoration and security.
  arXiv  Detail & Related papers  (2024-09-01T10:14:16Z)
• DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models [46.04803661300974]
  Large language models pretrained on a huge amount of data capture rich knowledge and information in the training data. The ability of data memorization and regurgitation in pretrained language models, revealed in previous studies, brings the risk of data leakage. We propose a framework DEPN to Detect and Edit Privacy Neurons in pretrained language models.
  arXiv  Detail & Related papers  (2023-10-31T03:09:36Z)
• Reconstructing Training Data from Model Gradient, Provably [68.21082086264555]
  We reconstruct the training samples from a single gradient query at a randomly chosen parameter value. As a provable attack that reveals sensitive training data, our findings suggest potential severe threats to privacy.
  arXiv  Detail & Related papers  (2022-12-07T15:32:22Z)
• Unintended memorisation of unique features in neural networks [15.174895411434026]
  We show that unique features occurring only once in training data are memorised by discriminative multi-layer perceptrons and convolutional neural networks. We develop a score estimating a model's sensitivity to a unique feature by comparing the KL divergences of the model's output distributions. We find that typical strategies to prevent overfitting do not prevent unique feature memorisation.
  arXiv  Detail & Related papers  (2022-05-20T10:48:18Z)
• Secure & Private Federated Neuroimaging [17.946206585229675]
  Federated Learning enables distributed training of neural network models over multiple data sources without sharing data. Each site trains the neural network over its private data for some time, then shares the neural network parameters with a Federation Controller. Our Federated Learning architecture, MetisFL, provides strong security and privacy.
  arXiv  Detail & Related papers  (2022-05-11T03:36:04Z)
• Mixed Differential Privacy in Computer Vision [133.68363478737058]
  AdaMix is an adaptive differentially private algorithm for training deep neural network classifiers using both private and public image data. A few-shot or even zero-shot learning baseline that ignores private data can outperform fine-tuning on a large private dataset.
  arXiv  Detail & Related papers  (2022-03-22T06:15:43Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• NeuralDP Differentially private neural networks by design [61.675604648670095]
  We propose NeuralDP, a technique for privatising activations of some layer within a neural network. We experimentally demonstrate on two datasets that our method offers substantially improved privacy-utility trade-offs compared to DP-SGD.
  arXiv  Detail & Related papers  (2021-07-30T12:40:19Z)
• Wide Network Learning with Differential Privacy [7.453881927237143]
  Current generation of neural networks suffers significant loss accuracy under most practically relevant privacy training regimes. We develop a general approach towards training these models that takes advantage of the sparsity of the gradients of private Empirical Minimization (ERM) Following the same number of parameters, we propose a novel algorithm for privately training neural networks.
  arXiv  Detail & Related papers  (2021-03-01T20:31:50Z)
• Robustness Threats of Differential Privacy [70.818129585404]
  We experimentally demonstrate that networks, trained with differential privacy, in some settings might be even more vulnerable in comparison to non-private versions. We study how the main ingredients of differentially private neural networks training, such as gradient clipping and noise addition, affect the robustness of the model.
  arXiv  Detail & Related papers  (2020-12-14T18:59:24Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.