Debias the Training of Diffusion Models

• URL: http://arxiv.org/abs/2310.08442v2
• Date: Sat, 4 Nov 2023 02:32:44 GMT
• Title: Debias the Training of Diffusion Models
• Authors: Hu Yu, Li Shen, Jie Huang, Man Zhou, Hongsheng Li, Feng Zhao
• Abstract summary: We provide theoretical evidence that the prevailing practice of using a constant loss weight strategy in diffusion models leads to biased estimation during the training phase. We propose an elegant and effective weighting strategy grounded in the theoretically unbiased principle. These analyses are expected to advance our understanding and demystify the inner workings of diffusion models.
• Score: 53.49637348771626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have demonstrated compelling generation quality by optimizing the variational lower bound through a simple denoising score matching loss. In this paper, we provide theoretical evidence that the prevailing practice of using a constant loss weight strategy in diffusion models leads to biased estimation during the training phase. Simply optimizing the denoising network to predict Gaussian noise with constant weighting may hinder precise estimations of original images. To address the issue, we propose an elegant and effective weighting strategy grounded in the theoretically unbiased principle. Moreover, we conduct a comprehensive and systematic exploration to dissect the inherent bias problem deriving from constant weighting loss from the perspectives of its existence, impact and reasons. These analyses are expected to advance our understanding and demystify the inner workings of diffusion models. Through empirical evaluation, we demonstrate that our proposed debiased estimation method significantly enhances sample quality without the reliance on complex techniques, and exhibits improved efficiency compared to the baseline method both in training and sampling processes.

Related papers

• Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models [24.186262549509102]
  This paper theoretically analyzes CFG in the context of masked discrete diffusion.<n>High guidance early in sampling (when inputs are heavily masked) harms generation quality, while late-stage guidance has a larger effect.<n>Our method smoothens the transport between the data distribution and the initial (masked/uniform) distribution, which results in improved sample quality.
  arXiv  Detail & Related papers  (2025-07-11T18:48:29Z)
• Deeper Diffusion Models Amplify Bias [46.2410852276839]
  Diffusion models may amplify inherent bias in the training data.<n>They may also compromise the presumed privacy of the training samples.<n>We introduce a training-free method to improve output quality in text-to-image and image-to-image generation.
  arXiv  Detail & Related papers  (2025-05-23T07:08:09Z)
• DIVE: Inverting Conditional Diffusion Models for Discriminative Tasks [79.50756148780928]
  This paper studies the problem of leveraging pretrained diffusion models for performing discriminative tasks. We extend the discriminative capability of pretrained frozen generative diffusion models from the classification task to the more complex object detection task, by "inverting" a pretrained layout-to-image diffusion model.
  arXiv  Detail & Related papers  (2025-04-24T05:13:27Z)
• Generalization through variance: how noise shapes inductive biases in diffusion models [0.0]
  We develop a mathematical theory that partly explains 'generalization through variance' phenomenon. We find that the distributions diffusion models effectively learn to sample from resemble their training distributions. We also characterize how this inductive bias interacts with feature-related inductive biases.
  arXiv  Detail & Related papers  (2025-04-16T23:41:10Z)
• InvDiff: Invariant Guidance for Bias Mitigation in Diffusion Models [28.51460282167433]
  diffusion models are highly data-driven and prone to inheriting imbalances and biases present in real-world data. We propose a framework, InvDiff, which aims to learn invariant semantic information for diffusion guidance. InvDiff effectively reduces biases while maintaining the quality of image generation.
  arXiv  Detail & Related papers  (2024-12-11T15:47:11Z)
• Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method [9.173055778539641]
  We propose a principled expectation-maximization (EM) framework that iteratively learns diffusion models from noisy data with arbitrary corruption types. Our framework employs a plug-and-play Monte Carlo method to accurately estimate clean images from noisy measurements, followed by training the diffusion model using the reconstructed images.
  arXiv  Detail & Related papers  (2024-10-15T03:54:59Z)
• Training-free Diffusion Model Alignment with Sampling Demons [15.400553977713914]
  We propose an optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions or model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through optimization. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models.
  arXiv  Detail & Related papers  (2024-10-08T07:33:49Z)
• Learning Diffusion Priors from Observations by Expectation Maximization [6.224769485481242]
  We present a novel method based on the expectation-maximization algorithm for training diffusion models from incomplete and noisy observations only. As part of our method, we propose and motivate an improved posterior sampling scheme for unconditional diffusion models.
  arXiv  Detail & Related papers  (2024-05-22T15:04:06Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models [52.1809084559048]
  We propose a novel two-stage divide-and-conquer training strategy termed TDC Training. It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models. While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• Eliminating Lipschitz Singularities in Diffusion Models [51.806899946775076]
  We show that diffusion models frequently exhibit the infinite Lipschitz near the zero point of timesteps. This poses a threat to the stability and accuracy of the diffusion process, which relies on integral operations. We propose a novel approach, dubbed E-TSDM, which eliminates the Lipschitz of the diffusion model near zero.
  arXiv  Detail & Related papers  (2023-06-20T03:05:28Z)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models [101.831766524264]
  Inverse tasks can be formulated as inferring a posterior distribution over data. This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. We propose a variational approach that by design seeks to approximate the true posterior distribution.
  arXiv  Detail & Related papers  (2023-05-07T23:00:47Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.