High Noise Scheduling is a Must

• URL: http://arxiv.org/abs/2404.06353v1
• Date: Tue, 9 Apr 2024 14:44:12 GMT
• Title: High Noise Scheduling is a Must
• Authors: Mahmut S. Gokmen, Cody Bumgardner, Jie Zhang, Ge Wang, Jin Chen,
• Abstract summary: Consistency models possess high capabilities for image generation, advancing sampling steps to a single step through their advanced techniques. Current advancements move one step forward consistency training techniques and eliminates the limitation of training. In this study, it is investigated the balance between high and low noise levels in noise distribution and offered noise distribution to maintain the stability.
• Score: 7.694256285730863
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Consistency models possess high capabilities for image generation, advancing sampling steps to a single step through their advanced techniques. Current advancements move one step forward consistency training techniques and eliminates the limitation of distillation training. Even though the proposed curriculum and noise scheduling in improved training techniques yield better results than basic consistency models, it lacks well balanced noise distribution and its consistency between curriculum. In this study, it is investigated the balance between high and low noise levels in noise distribution and offered polynomial noise distribution to maintain the stability. This proposed polynomial noise distribution is also supported with a predefined Karras noises to prevent unique noise levels arises with Karras noise generation algorithm. Furthermore, by elimination of learned noisy steps with a curriculum based on sinusoidal function increase the performance of the model in denoising. To make a fair comparison with the latest released consistency model training techniques, experiments are conducted with same hyper-parameters except curriculum and noise distribution. The models utilized during experiments are determined with low depth to prove the robustness of our proposed technique. The results show that the polynomial noise distribution outperforms the model trained with log-normal noise distribution, yielding a 33.54 FID score after 100,000 training steps with constant discretization steps. Additionally, the implementation of a sinusoidal-based curriculum enhances denoising performance, resulting in a FID score of 30.48.

Related papers

• Improved Noise Schedule for Diffusion Training [51.849746576387375]
  We propose a novel approach to design the noise schedule for enhancing the training of diffusion models. We empirically demonstrate the superiority of our noise schedule over the standard cosine schedule.
  arXiv  Detail & Related papers  (2024-07-03T17:34:55Z)
• Bayesian Inference of General Noise Model Parameters from Surface Code's Syndrome Statistics [0.0]
  We propose general noise model Bayesian inference methods that integrate the surface code's tensor network simulator. For stationary noise, where the noise parameters are constant and do not change, we propose a method based on the Markov chain Monte Carlo. For time-varying noise, which is a more realistic situation, we introduce another method based on the sequential Monte Carlo.
  arXiv  Detail & Related papers  (2024-06-13T10:26:04Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Blue noise for diffusion models [50.99852321110366]
  We introduce a novel and general class of diffusion models taking correlated noise within and across images into account. Our framework allows introducing correlation across images within a single mini-batch to improve gradient flow. We perform both qualitative and quantitative evaluations on a variety of datasets using our method.
  arXiv  Detail & Related papers  (2024-02-07T14:59:25Z)
• Not All Steps are Equal: Efficient Generation with Progressive Diffusion Models [62.155612146799314]
  We propose a novel two-stage training strategy termed Step-Adaptive Training. In the initial stage, a base denoising model is trained to encompass all timesteps. We partition the timesteps into distinct groups, fine-tuning the model within each group to achieve specialized denoising capabilities.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• May the Noise be with you: Adversarial Training without Adversarial Examples [3.4673556247932225]
  We investigate the question: Can we obtain adversarially-trained models without training on adversarial? Our proposed approach incorporates inherentity by embedding Gaussian noise within the layers of the NN model at training time. Our work contributes adversarially trained networks using a completely different approach, with empirically similar robustness to adversarial training.
  arXiv  Detail & Related papers  (2023-12-12T08:22:28Z)
• Realistic Noise Synthesis with Diffusion Models [68.48859665320828]
  Deep image denoising models often rely on large amount of training data for the high quality performance. We propose a novel method that synthesizes realistic noise using diffusion models, namely Realistic Noise Synthesize Diffusor (RNSD) RNSD can incorporate guided multiscale content, such as more realistic noise with spatial correlations can be generated at multiple frequencies.
  arXiv  Detail & Related papers  (2023-05-23T12:56:01Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Self-Adapting Noise-Contrastive Estimation for Energy-Based Models [0.0]
  Training energy-based models with noise-contrastive estimation (NCE) is theoretically feasible but practically challenging. Previous works have explored modelling the noise distribution as a separate generative model, and then concurrently training this noise model with the EBM. This thesis proposes a self-adapting NCE algorithm which uses static instances of the EBM along its training trajectory as the noise distribution.
  arXiv  Detail & Related papers  (2022-11-03T15:17:43Z)
• The Optimal Noise in Noise-Contrastive Learning Is Not What You Think [80.07065346699005]
  We show that deviating from this assumption can actually lead to better statistical estimators. In particular, the optimal noise distribution is different from the data's and even from a different family.
  arXiv  Detail & Related papers  (2022-03-02T13:59:20Z)
• Denoising Distantly Supervised Named Entity Recognition via a Hypergeometric Probabilistic Model [26.76830553508229]
  Hypergeometric Learning (HGL) is a denoising algorithm for distantly supervised named entity recognition. HGL takes both noise distribution and instance-level confidence into consideration. Experiments show that HGL can effectively denoise the weakly-labeled data retrieved from distant supervision.
  arXiv  Detail & Related papers  (2021-06-17T04:01:25Z)
• Noise Estimation for Generative Diffusion Models [91.22679787578438]
  In this work, we present a simple and versatile learning scheme that can adjust the noise parameters for any given number of steps. Our approach comes at a negligible computation cost.
  arXiv  Detail & Related papers  (2021-04-06T15:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.