Pre-trained Perceptual Features Improve Differentially Private Image Generation

• URL: http://arxiv.org/abs/2205.12900v4
• Date: Thu, 20 Jul 2023 12:10:09 GMT
• Title: Pre-trained Perceptual Features Improve Differentially Private Image Generation
• Authors: Fredrik Harder and Milad Jalali Asadabadi and Danica J. Sutherland and Mijung Park
• Abstract summary: Training even moderately-sized generative models with differentially-private descent gradient (DP-SGD) is difficult. We advocate building off a good, relevant representation on an informative public dataset, then learning to model the private data with that representation. Our work introduces simple yet powerful foundations for reducing the gap between private and non-private deep generative models.
• Score: 8.659595986100738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training even moderately-sized generative models with differentially-private stochastic gradient descent (DP-SGD) is difficult: the required level of noise for reasonable levels of privacy is simply too large. We advocate instead building off a good, relevant representation on an informative public dataset, then learning to model the private data with that representation. In particular, we minimize the maximum mean discrepancy (MMD) between private target data and a generator's distribution, using a kernel based on perceptual features learned from a public dataset. With the MMD, we can simply privatize the data-dependent term once and for all, rather than introducing noise at each step of optimization as in DP-SGD. Our algorithm allows us to generate CIFAR10-level images with $\epsilon \approx 2$ which capture distinctive features in the distribution, far surpassing the current state of the art, which mostly focuses on datasets such as MNIST and FashionMNIST at a large $\epsilon \approx 10$. Our work introduces simple yet powerful foundations for reducing the gap between private and non-private deep generative models. Our code is available at \url{https://github.com/ParkLabML/DP-MEPF}.

Related papers

• Private Fine-tuning of Large Language Models with Zeroth-order Optimization [51.19403058739522]
  Differentially private gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner. We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods.
  arXiv  Detail & Related papers  (2024-01-09T03:53:59Z)
• Learning to Generate Image Embeddings with User-level Differential Privacy [31.797461992234457]
  DP-FedEmb is a variant of federated learning algorithms with per-user sensitivity control and noise addition. We show it is possible to achieve strong user-level DP guarantees of $epsilon4$ while controlling the utility drop within 5%, when millions of users can participate in training.
  arXiv  Detail & Related papers  (2022-11-20T01:59:37Z)
• TAN Without a Burn: Scaling Laws of DP-SGD [70.7364032297978]
  Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently. We decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements. We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in top-1 accuracy.
  arXiv  Detail & Related papers  (2022-10-07T08:44:35Z)
• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent [69.14164921515949]
  We characterize privacy guarantees for individual examples when releasing models trained by DP-SGD. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. This implies groups that are underserved in terms of model utility simultaneously experience weaker privacy guarantees.
  arXiv  Detail & Related papers  (2022-06-06T13:49:37Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• Mixed Differential Privacy in Computer Vision [133.68363478737058]
  AdaMix is an adaptive differentially private algorithm for training deep neural network classifiers using both private and public image data. A few-shot or even zero-shot learning baseline that ignores private data can outperform fine-tuning on a large private dataset.
  arXiv  Detail & Related papers  (2022-03-22T06:15:43Z)
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence [73.14373832423156]
  We propose DP-Sinkhorn, a novel optimal transport-based generative method for learning data distributions from private data with differential privacy. Unlike existing approaches for training differentially private generative models, we do not rely on adversarial objectives.
  arXiv  Detail & Related papers  (2021-11-01T18:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.