Toward Training at ImageNet Scale with Differential Privacy

• URL: http://arxiv.org/abs/2201.12328v1
• Date: Fri, 28 Jan 2022 18:48:18 GMT
• Title: Toward Training at ImageNet Scale with Differential Privacy
• Authors: Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, Abhradeep Thakurta
• Abstract summary: Differential privacy (DP) is the de facto standard for training machine learning (ML) models. ImageNet image classification is a poster example of an ML task that is very challenging to resolve accurately with DP. This paper shares initial lessons from our effort, in the hope that it will inspire and inform other researchers to explore DP training at scale.
• Score: 19.139956067438995
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differential privacy (DP) is the de facto standard for training machine learning (ML) models, including neural networks, while ensuring the privacy of individual examples in the training set. Despite a rich literature on how to train ML models with differential privacy, it remains extremely challenging to train real-life, large neural networks with both reasonable accuracy and privacy. We set out to investigate how to do this, using ImageNet image classification as a poster example of an ML task that is very challenging to resolve accurately with DP right now. This paper shares initial lessons from our effort, in the hope that it will inspire and inform other researchers to explore DP training at scale. We show approaches which help to make DP training faster, as well as model types and settings of the training process that tend to work better for DP. Combined, the methods we discuss let us train a Resnet-18 with differential privacy to 47.9% accuracy and privacy parameters $\epsilon = 10, \delta = 10^{-6}$, a significant improvement over "naive" DP-SGD training of Imagenet models but a far cry from the $75\%$ accuracy that can be obtained by the same network without privacy. We share our code at https://github.com/google-research/dp-imagenet calling for others to join us in moving the needle further on DP at scale.

Related papers

• Differentially Private Representation Learning via Image Captioning [51.45515227171524]
  We show that effective DP representation learning can be done via image captioning and scaling up to internet-scale multimodal datasets. We successfully train a DP image captioner (DP-Cap) on a 233M subset of LAION-2B from scratch using a reasonable amount of computation.
  arXiv  Detail & Related papers  (2024-03-04T21:52:25Z)
• Efficient Verification-Based Face Identification [50.616875565173274]
  We study the problem of performing face verification with an efficient neural model $f$. Our model leads to a substantially small $f$ requiring only 23k parameters and 5M floating point operations (FLOPS) We use six face verification datasets to demonstrate that our method is on par or better than state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-20T18:08:02Z)
• MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments [72.6405488990753]
  Self-supervised learning can be used for mitigating the greedy needs of Vision Transformer networks. We propose a single-stage and standalone method, MOCA, which unifies both desired properties. We achieve new state-of-the-art results on low-shot settings and strong experimental results in various evaluation protocols.
  arXiv  Detail & Related papers  (2023-07-18T15:46:20Z)
• Differentially Private Image Classification by Learning Priors from Random Processes [48.0766422536737]
  In privacy-preserving machine learning, differentially private gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data.
  arXiv  Detail & Related papers  (2023-06-08T04:14:32Z)
• Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models [7.49320945341034]
  We show that small and efficient architecture design can outperform current state-of-the-art models with substantially lower computational requirements. Our results are a step towards efficient model architectures that make optimal use of their parameters.
  arXiv  Detail & Related papers  (2023-01-30T17:43:47Z)
• TAN Without a Burn: Scaling Laws of DP-SGD [70.7364032297978]
  Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently. We decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements. We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in top-1 accuracy.
  arXiv  Detail & Related papers  (2022-10-07T08:44:35Z)
• Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search [38.83524780461911]
  We show how carefully selecting the layers being fine-tuned in the pretrained neural network allows us to establish new state-of-the-art tradeoffs between privacy and accuracy. We achieve 77.9% accuracy for $(varepsilon, delta)= (2, 10-5)$ on CIFAR-100 for a model pretrained on ImageNet.
  arXiv  Detail & Related papers  (2022-10-05T11:32:49Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.