Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation

• URL: http://arxiv.org/abs/2408.14738v1
• Date: Tue, 27 Aug 2024 02:29:29 GMT
• Title: Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation
• Authors: Bochao Liu, Pengju Wang, Shiming Ge,
• Abstract summary: We introduce DP-SAD, which trains a private diffusion model by an adversarial distillation method. For better generation quality, we introduce a discriminator to distinguish whether an image is from the teacher or the student.
• Score: 20.62325580203137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While the success of deep learning relies on large amounts of training datasets, data is often limited in privacy-sensitive domains. To address this challenge, generative model learning with differential privacy has emerged as a solution to train private generative models for desensitized data generation. However, the quality of the images generated by existing methods is limited due to the complexity of modeling data distribution. We build on the success of diffusion models and introduce DP-SAD, which trains a private diffusion model by a stochastic adversarial distillation method. Specifically, we first train a diffusion model as a teacher and then train a student by distillation, in which we achieve differential privacy by adding noise to the gradients from other models to the student. For better generation quality, we introduce a discriminator to distinguish whether an image is from the teacher or the student, which forms the adversarial training. Extensive experiments and analysis clearly demonstrate the effectiveness of our proposed method.

Related papers

• Constrained Diffusion Models via Dual Training [80.03953599062365]
  Diffusion processes are prone to generating samples that reflect biases in a training dataset. We develop constrained diffusion models by imposing diffusion constraints based on desired distributions. We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
  arXiv  Detail & Related papers  (2024-08-27T14:25:42Z)
• Gradient Inversion of Federated Diffusion Models [4.1355611383748005]
  Diffusion models are becoming defector generative models, which generate exceptionally high-resolution image data. In this paper, we study the privacy risk of gradient inversion attacks. We propose a triple-optimization GIDM+ that coordinates the optimization of the unknown data.
  arXiv  Detail & Related papers  (2024-05-30T18:00:03Z)
• Training Class-Imbalanced Diffusion Model Via Overlap Optimization [55.96820607533968]
  Diffusion models trained on real-world datasets often yield inferior fidelity for tail classes. Deep generative models, including diffusion models, are biased towards classes with abundant training images. We propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes.
  arXiv  Detail & Related papers  (2024-02-16T16:47:21Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Expanding Expressiveness of Diffusion Models with Limited Data via Self-Distillation based Fine-Tuning [24.791783885165923]
  Training diffusion models on limited datasets poses challenges in terms of limited generation capacity and expressiveness. We propose Self-Distillation for Fine-Tuning diffusion models (SDFT) to address these challenges.
  arXiv  Detail & Related papers  (2023-11-02T06:24:06Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• Phoenix: A Federated Generative Diffusion Model [6.09170287691728]
  Training generative models on large centralized datasets can pose challenges in terms of data privacy, security, and accessibility. This paper proposes a novel method for training a Denoising Diffusion Probabilistic Model (DDPM) across multiple data sources using Federated Learning (FL) techniques.
  arXiv  Detail & Related papers  (2023-06-07T01:43:09Z)
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models [77.83923746319498]
  We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models. We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
  arXiv  Detail & Related papers  (2023-05-29T04:22:57Z)
• Private Gradient Estimation is Useful for Generative Modeling [25.777591229903596]
  We present a new private generative modeling approach where samples are generated via Hamiltonian dynamics with gradients of the private dataset estimated by a well-trained network. Our model is able to generate data with a resolution of 256x256.
  arXiv  Detail & Related papers  (2023-05-18T02:51:17Z)
• Extracting Training Data from Diffusion Models [77.11719063152027]
  We show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models. We train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy.
  arXiv  Detail & Related papers  (2023-01-30T18:53:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.