Efficient Differentially Private Fine-Tuning of Diffusion Models

• URL: http://arxiv.org/abs/2406.05257v1
• Date: Fri, 7 Jun 2024 21:00:20 GMT
• Title: Efficient Differentially Private Fine-Tuning of Diffusion Models
• Authors: Jing Liu, Andrew Lowy, Toshiaki Koike-Akino, Kieran Parsons, Ye Wang,
• Abstract summary: Fine-tuning large diffusion models with DP-SGD can be very resource-demanding in terms of memory usage and computation. In this work, we investigate Efficient Fine-Tuning (PEFT) of diffusion models using Low-Dimensional Adaptation (LoDA) with Differential Privacy. Our source code will be made available on GitHub.
• Score: 15.71777343534365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent developments of Diffusion Models (DMs) enable generation of astonishingly high-quality synthetic samples. Recent work showed that the synthetic samples generated by the diffusion model, which is pre-trained on public data and fully fine-tuned with differential privacy on private data, can train a downstream classifier, while achieving a good privacy-utility tradeoff. However, fully fine-tuning such large diffusion models with DP-SGD can be very resource-demanding in terms of memory usage and computation. In this work, we investigate Parameter-Efficient Fine-Tuning (PEFT) of diffusion models using Low-Dimensional Adaptation (LoDA) with Differential Privacy. We evaluate the proposed method with the MNIST and CIFAR-10 datasets and demonstrate that such efficient fine-tuning can also generate useful synthetic samples for training downstream classifiers, with guaranteed privacy protection of fine-tuning data. Our source code will be made available on GitHub.

Related papers

• Data Augmentation via Diffusion Model to Enhance AI Fairness [1.2979015577834876]
  This paper explores the potential of diffusion models to generate synthetic data to improve AI fairness. The Tabular Denoising Diffusion Probabilistic Model (Tab-DDPM) was utilized with different amounts of generated data for data augmentation. Experimental results demonstrate that the synthetic data generated by Tab-DDPM improves fairness in binary classification.
  arXiv  Detail & Related papers  (2024-10-20T18:52:31Z)
• Constrained Diffusion Models via Dual Training [80.03953599062365]
  Diffusion processes are prone to generating samples that reflect biases in a training dataset. We develop constrained diffusion models by imposing diffusion constraints based on desired distributions. We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
  arXiv  Detail & Related papers  (2024-08-27T14:25:42Z)
• Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment [76.44483062571611]
  Test-time adaptation (TTA) aims to enhance the performance of source-domain pretrained models when tested on unknown shifted target domains. Traditional TTA methods primarily adapt model weights based on target data streams, making model performance sensitive to the amount and order of target data. Recent diffusion-driven TTA methods have demonstrated strong performance by using an unconditional diffusion model.
  arXiv  Detail & Related papers  (2024-06-06T17:39:09Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Private Synthetic Data Meets Ensemble Learning [15.425653946755025]
  When machine learning models are trained on synthetic data and then deployed on real data, there is often a performance drop. We introduce a new ensemble strategy for training downstream models, with the goal of enhancing their performance when used on real data.
  arXiv  Detail & Related papers  (2023-10-15T04:24:42Z)
• PriSampler: Mitigating Property Inference of Diffusion Models [6.5990719141691825]
  This work systematically presents the first privacy study about property inference attacks against diffusion models. We propose a new model-agnostic plug-in method PriSampler to infer the risks of the property inference of diffusion models.
  arXiv  Detail & Related papers  (2023-06-08T14:05:06Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Synthesizing Mixed-type Electronic Health Records using Diffusion Models [10.973115905786129]
  Synthetic data generation is a promising solution to mitigate privacy concerns when sharing sensitive patient information. Recent studies have shown that diffusion models offer several advantages over GANs, such as generation of more realistic synthetic data and stable training in generating data modalities, including image, text, and sound. Our experiments demonstrate that TabDDPM outperforms the state-of-the-art models across all evaluation metrics, except for privacy, which confirms the trade-off between privacy and utility.
  arXiv  Detail & Related papers  (2023-02-28T15:42:30Z)
• Differentially Private Diffusion Models Generate Useful Synthetic Images [53.94025967603649]
  Recent studies have found that, by default, the outputs of some diffusion models do not preserve training data privacy. By privately fine-tuning ImageNet pre-trained diffusion models with more than 80M parameters, we obtain SOTA results on CIFAR-10 and Camelyon17. Our results demonstrate that diffusion models fine-tuned with differential privacy can produce useful and provably private synthetic data.
  arXiv  Detail & Related papers  (2023-02-27T15:02:04Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.