What happens to diffusion model likelihood when your model is conditional?

• URL: http://arxiv.org/abs/2409.06364v2
• Date: Thu, 26 Sep 2024 12:33:46 GMT
• Title: What happens to diffusion model likelihood when your model is conditional?
• Authors: Mattias Cross, Anton Ragni,
• Abstract summary: Diffusion Models (DMs) iteratively denoise random samples to produce high-quality data. DMs have been used to rank unconditional DMs and for out-of-domain classification. We show that applying DMs to conditional tasks reveals inconsistencies and strengthens claims that the properties of DM likelihood are unknown.
• Score: 1.643629306994231
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Models (DMs) iteratively denoise random samples to produce high-quality data. The iterative sampling process is derived from Stochastic Differential Equations (SDEs), allowing a speed-quality trade-off chosen at inference. Another advantage of sampling with differential equations is exact likelihood computation. These likelihoods have been used to rank unconditional DMs and for out-of-domain classification. Despite the many existing and possible uses of DM likelihoods, the distinct properties captured are unknown, especially in conditional contexts such as Text-To-Image (TTI) or Text-To-Speech synthesis (TTS). Surprisingly, we find that TTS DM likelihoods are agnostic to the text input. TTI likelihood is more expressive but cannot discern confounding prompts. Our results show that applying DMs to conditional tasks reveals inconsistencies and strengthens claims that the properties of DM likelihood are unknown. This impact sheds light on the previously unknown nature of DM likelihoods. Although conditional DMs maximise likelihood, the likelihood in question is not as sensitive to the conditioning input as one expects. This investigation provides a new point-of-view on diffusion likelihoods.

Related papers

• Training-Free Safe Denoisers for Safe Use of Diffusion Models [49.045799120267915]
  Powerful diffusion models (DMs) are often misused to generate not-safe-for-work (NSFW) content or generate copyrighted material or data of individuals who wish to be forgotten. We develop a practical algorithm that produces high-quality samples while avoiding negation areas of the data distribution. These results hint at the great potential of our training-free safe denoiser for using DMs more safely.
  arXiv  Detail & Related papers  (2025-02-11T23:14:39Z)
• Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images? [21.600998338094794]
  We focus on the ability of diffusion models (DMs) to learn hidden rules between image features. We investigate whether DMs can accurately capture the inter-feature rule ($p(mathbfy|mathbfx)$) We design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities.
  arXiv  Detail & Related papers  (2025-02-07T07:49:37Z)
• Improving image synthesis with diffusion-negative sampling [54.84368884047812]
  We propose a new diffusion-negative prompting (DNP) strategy for image generation with diffusion models (DMs) DNP is based on a new procedure to sample images that are least compliant with p under the distribution of the DM, denoted as diffusion-negative sampling (DNS) DNS is straightforward to implement and requires no training. Experiments and human evaluations show that DNP performs well both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2024-11-08T10:58:09Z)
• Slight Corruption in Pre-training Data Makes Better Diffusion Models [71.90034201302397]
  Diffusion models (DMs) have shown remarkable capabilities in generating high-quality images, audios, and videos. DMs benefit significantly from extensive pre-training on large-scale datasets. However, pre-training datasets often contain corrupted pairs where conditions do not accurately describe the data. This paper presents the first comprehensive study on the impact of such corruption in pre-training data of DMs.
  arXiv  Detail & Related papers  (2024-05-30T21:35:48Z)
• Don't drop your samples! Coherence-aware training benefits Conditional diffusion [17.349357521783062]
  Coherence-Aware Diffusion (CAD) is a novel method that integrates coherence in conditional information into diffusion models. We show that CAD is theoretically sound and empirically effective on various conditional generation tasks.
  arXiv  Detail & Related papers  (2024-05-30T17:57:26Z)
• On the Asymptotic Mean Square Error Optimality of Diffusion Models [10.72484143420088]
  Diffusion models (DMs) as generative priors have recently shown great potential for denoising tasks. This paper proposes a novel denoising strategy inspired by the structure of the MSE-optimal conditional mean (CME) The resulting DM-based denoiser can be conveniently employed using a pre-trained DM, being particularly fast by truncating reverse diffusion steps.
  arXiv  Detail & Related papers  (2024-03-05T13:25:44Z)
• ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbation [18.103478658038846]
  Text-to-Image (T2I) Diffusion Models (DMs) have shown impressive abilities in generating high-quality images based on simple text descriptions. As is common with many Deep Learning (DL) models, DMs are subject to a lack of robustness. We introduce a probabilistic notion of T2I DMs' robustness; and then establish an efficient framework, ProTIP, to evaluate it with statistical guarantees.
  arXiv  Detail & Related papers  (2024-02-23T16:48:56Z)
• Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand [9.460857822923842]
  Causal inference from observational data plays critical role in many applications in trustworthy machine learning. We show how to sample from any identifiable interventional distribution given an arbitrary causal graph. We also generate high-dimensional interventional samples from the MIMIC-CXR dataset involving text and image variables.
  arXiv  Detail & Related papers  (2024-02-12T05:48:31Z)
• Risk-Sensitive Diffusion: Robustly Optimizing Diffusion Models with Noisy Samples [58.68233326265417]
  Non-image data are prevalent in real applications and tend to be noisy. Risk-sensitive SDE is a type of differential equation (SDE) parameterized by the risk vector. We conduct systematic studies for both Gaussian and non-Gaussian noise distributions.
  arXiv  Detail & Related papers  (2024-02-03T08:41:51Z)
• Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllability [62.105715985563656]
  We propose a novel framework dubbed Diffusion-Based Projected Gradient Descent (Diff-PGD) for generating realistic adversarial samples. Our framework can be easily customized for specific tasks such as digital attacks, physical-world attacks, and style-based attacks.
  arXiv  Detail & Related papers  (2023-05-25T21:51:23Z)
• Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score [62.54911162109439]
  Adversarial detection aims to determine whether a given sample is an adversarial one based on the discrepancy between natural and adversarial distributions. We propose a new statistic called expected perturbation score (EPS), which is essentially the expected score of a sample after various perturbations. We develop EPS-based maximum mean discrepancy (MMD) as a metric to measure the discrepancy between the test sample and natural samples.
  arXiv  Detail & Related papers  (2023-05-25T13:14:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.