Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization

• URL: http://arxiv.org/abs/2407.14041v2
• Date: Sat, 27 Jul 2024 14:22:56 GMT
• Title: Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization
• Authors: Zipeng Qi, Lichen Bai, Haoyi Xiong, Zeke Xie,
• Abstract summary: Diffusion models can generate high-quality data from randomly sampled Gaussian noises. We show that not all noises are created equally for diffusion models. We propose a novel noise optimization method that actively enhances the inversion of arbitrary noises.
• Score: 23.795237240203456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models that can generate high-quality data from randomly sampled Gaussian noises have become the mainstream generative method in both academia and industry. Are randomly sampled Gaussian noises equally good for diffusion models? While a large body of works tried to understand and improve diffusion models, previous works overlooked the possibility to select or optimize the sampled noise the possibility of selecting or optimizing sampled noises for improving diffusion models. In this paper, we mainly made three contributions. First, we report that not all noises are created equally for diffusion models. We are the first to hypothesize and empirically observe that the generation quality of diffusion models significantly depend on the noise inversion stability. This naturally provides us a noise selection method according to the inversion stability. Second, we further propose a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. Our method is the first one that works on noise space to generally improve generated results without fine-tuning diffusion models. Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo, in terms of human preference and other objective evaluation metrics. For example, the human preference winning rates of noise selection and noise optimization over the baselines can be up to 57% and 72.5%, respectively, on DrawBench.

Related papers

• Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Model [7.194019884532405]
  ANSE is a model-aware framework that selects high-quality noise seeds by quantifying attention-based uncertainty.<n>Experiments on CogVideoX-2B and 5B demonstrate that ANSE improves video quality with only an 8% and 13% increase in inference time.
  arXiv  Detail & Related papers  (2025-05-23T07:09:10Z)
• Arbitrary-steps Image Super-resolution via Diffusion Inversion [68.78628844966019]
  This study presents a new image super-resolution (SR) technique based on diffusion inversion, aiming at harnessing the rich image priors encapsulated in large pre-trained diffusion models to improve SR performance. We design a Partial noise Prediction strategy to construct an intermediate state of the diffusion model, which serves as the starting sampling point. Once trained, this noise predictor can be used to initialize the sampling process partially along the diffusion trajectory, generating the desirable high-resolution result.
  arXiv  Detail & Related papers  (2024-12-12T07:24:13Z)
• Diffusion Gaussian Mixture Audio Denoise [23.760755498636943]
  We propose a DiffGMM model, a denoising model based on the diffusion and Gaussian mixture models. Given a noisy audio signal, we first apply a 1D-U-Net to extract features and train linear layers to estimate parameters for the Gaussian mixture model. The noisy signal is continuously subtracted from the estimated noise to output clean audio signals.
  arXiv  Detail & Related papers  (2024-06-13T14:18:10Z)
• Blue noise for diffusion models [50.99852321110366]
  We introduce a novel and general class of diffusion models taking correlated noise within and across images into account. Our framework allows introducing correlation across images within a single mini-batch to improve gradient flow. We perform both qualitative and quantitative evaluations on a variety of datasets using our method.
  arXiv  Detail & Related papers  (2024-02-07T14:59:25Z)
• Robust Estimation of Causal Heteroscedastic Noise Models [7.568978862189266]
  Student's $t$-distribution is known for its robustness in accounting for sampling variability with smaller sample sizes and extreme values without significantly altering the overall distribution shape. Our empirical evaluations demonstrate that our estimators are more robust and achieve better overall performance across synthetic and real benchmarks.
  arXiv  Detail & Related papers  (2023-12-15T02:26:35Z)
• Multi-scale Diffusion Denoised Smoothing [79.95360025953931]
  randomized smoothing has become one of a few tangible approaches that offers adversarial robustness to models at scale. We present scalable methods to address the current trade-off between certified robustness and accuracy in denoised smoothing. Our experiments show that the proposed multi-scale smoothing scheme combined with diffusion fine-tuning enables strong certified robustness available with high noise level.
  arXiv  Detail & Related papers  (2023-10-25T17:11:21Z)
• Denoising Diffusion Gamma Models [91.22679787578438]
  We introduce the Denoising Diffusion Gamma Model (DDGM) and show that noise from Gamma distribution provides improved results for image and speech generation. Our approach preserves the ability to efficiently sample state in the training diffusion process while using Gamma noise.
  arXiv  Detail & Related papers  (2021-10-10T10:46:31Z)
• Rethinking Noise Synthesis and Modeling in Raw Denoising [75.55136662685341]
  We introduce a new perspective to synthesize noise by directly sampling from the sensor's real noise. It inherently generates accurate raw image noise for different camera sensors.
  arXiv  Detail & Related papers  (2021-10-10T10:45:24Z)
• A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
  We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals. The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
  arXiv  Detail & Related papers  (2021-07-25T19:23:18Z)
• Non Gaussian Denoising Diffusion Models [91.22679787578438]
  We show that noise from Gamma distribution provides improved results for image and speech generation. We also show that using a mixture of Gaussian noise variables in the diffusion process improves the performance over a diffusion process that is based on a single distribution.
  arXiv  Detail & Related papers  (2021-06-14T16:42:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.