Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy

• URL: http://arxiv.org/abs/2405.14800v3
• Date: Sun, 27 Oct 2024 12:43:56 GMT
• Title: Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy
• Authors: Shengfang Zhai, Huanran Chen, Yinpeng Dong, Jiajun Li, Qingni Shen, Yansong Gao, Hang Su, Yang Liu,
• Abstract summary: We propose a conditional overfitting phenomenon in text-to-image diffusion models. Our method significantly outperforms previous methods across various data and dataset scales.
• Score: 36.156856772794065
• License:
• Abstract: Text-to-image diffusion models have achieved tremendous success in the field of controllable image generation, while also coming along with issues of privacy leakage and data copyrights. Membership inference arises in these contexts as a potential auditing method for detecting unauthorized data usage. While some efforts have been made on diffusion models, they are not applicable to text-to-image diffusion models due to the high computation overhead and enhanced generalization capabilities. In this paper, we first identify a conditional overfitting phenomenon in text-to-image diffusion models, indicating that these models tend to overfit the conditional distribution of images given the corresponding text rather than the marginal distribution of images only. Based on this observation, we derive an analytical indicator, namely Conditional Likelihood Discrepancy (CLiD), to perform membership inference, which reduces the stochasticity in estimating memorization of individual samples. Experimental results demonstrate that our method significantly outperforms previous methods across various data distributions and dataset scales. Additionally, our method shows superior resistance to overfitting mitigation strategies, such as early stopping and data augmentation.

Related papers

• Fast constrained sampling in pre-trained diffusion models [77.21486516041391]
  Diffusion models have dominated the field of large, generative image models. We propose an algorithm for fast-constrained sampling in large pre-trained diffusion models.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method [9.173055778539641]
  We propose a principled expectation-maximization (EM) framework that iteratively learns diffusion models from noisy data with arbitrary corruption types. Our framework employs a plug-and-play Monte Carlo method to accurately estimate clean images from noisy measurements, followed by training the diffusion model using the reconstructed images.
  arXiv  Detail & Related papers  (2024-10-15T03:54:59Z)
• Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering [15.326641037243006]
  diffusion models can effectively learn the image distribution and generate new samples. We provide theoretical insights into this phenomenon by leveraging key empirical observations. We show that the minimal number of samples required to learn the underlying distribution scales linearly with the intrinsic dimensions.
  arXiv  Detail & Related papers  (2024-09-04T04:14:02Z)
• Integrating Amortized Inference with Diffusion Models for Learning Clean Distribution from Corrupted Images [19.957503854446735]
  Diffusion models (DMs) have emerged as powerful generative models for solving inverse problems. FlowDiff is a joint training paradigm that leverages a conditional normalizing flow model to facilitate the training of diffusion models on corrupted data sources. Our experiment shows that FlowDiff can effectively learn clean distributions across a wide range of corrupted data sources.
  arXiv  Detail & Related papers  (2024-07-15T18:33:20Z)
• Adversarial Examples are Misaligned in Diffusion Model Manifolds [7.979892202477701]
  This study is dedicated to the investigation of adversarial attacks through the lens of diffusion models. Our focus lies in utilizing the diffusion model to detect and analyze the anomalies introduced by these attacks on images. Results demonstrate a notable capacity to discriminate effectively between benign and attacked images.
  arXiv  Detail & Related papers  (2024-01-12T15:29:21Z)
• The Journey, Not the Destination: How Data Guides Diffusion Models [75.19694584942623]
  Diffusion models trained on large datasets can synthesize photo-realistic images of remarkable quality and diversity. We propose a framework that: (i) provides a formal notion of data attribution in the context of diffusion models, and (ii) allows us to counterfactually validate such attributions.
  arXiv  Detail & Related papers  (2023-12-11T08:39:43Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction [75.91471250967703]
  We introduce a novel sampling framework called Steerable Conditional Diffusion. This framework adapts the diffusion model, concurrently with image reconstruction, based solely on the information provided by the available measurement. We achieve substantial enhancements in out-of-distribution performance across diverse imaging modalities.
  arXiv  Detail & Related papers  (2023-08-28T08:47:06Z)
• Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners [88.07317175639226]
  We propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach mainly uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information.
  arXiv  Detail & Related papers  (2023-05-18T05:41:36Z)
• Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors [64.24948495708337]
  We introduce a new method that brings predicted samples to the training data manifold using a pretrained unconditional diffusion model. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks.
  arXiv  Detail & Related papers  (2022-12-14T17:26:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.