An analysis of the noise schedule for score-based generative models

• URL: http://arxiv.org/abs/2402.04650v4
• Date: Mon, 27 Jan 2025 10:43:16 GMT
• Title: An analysis of the noise schedule for score-based generative models
• Authors: Stanislas Strasman, Antonio Ocello, Claire Boyer, Sylvain Le Corff, Vincent Lemaire,
• Abstract summary: Score-based generative models (SGMs) aim at estimating a target data distribution by learning score functions using only noise-perturbed samples from the target. Recent literature has focused extensively on assessing the error between the target and estimated distributions, gauging the generative quality through the Kullback-Leibler (KL) divergence and Wasserstein distances. We establish an upper bound for the KL divergence between the target and the estimated distributions, explicitly depending on any time-dependent noise schedule.
• Score: 7.180235086275926
• License:
• Abstract: Score-based generative models (SGMs) aim at estimating a target data distribution by learning score functions using only noise-perturbed samples from the target.Recent literature has focused extensively on assessing the error between the target and estimated distributions, gauging the generative quality through the Kullback-Leibler (KL) divergence and Wasserstein distances. Under mild assumptions on the data distribution, we establish an upper bound for the KL divergence between the target and the estimated distributions, explicitly depending on any time-dependent noise schedule. Under additional regularity assumptions, taking advantage of favorable underlying contraction mechanisms, we provide a tighter error bound in Wasserstein distance compared to state-of-the-art results. In addition to being tractable, this upper bound jointly incorporates properties of the target distribution and SGM hyperparameters that need to be tuned during training. Finally, we illustrate these bounds through numerical experiments using simulated and CIFAR-10 datasets, identifying an optimal range of noise schedules within a parametric family.

Related papers

• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models [11.743167854433306]
  Diffusion models generate samples by estimating the score function of the target distribution at various noise levels. In this work, we establish the first (nearly) dimension-free sample bounds complexity for learning these score functions. A key aspect of our analysis is the use of a single function approximator to jointly estimate scores across noise levels.
  arXiv  Detail & Related papers  (2025-02-14T18:32:22Z)
• Beyond Log-Concavity and Score Regularity: Improved Convergence Bounds for Score-Based Generative Models in W2-distance [0.0]
  We present a novel framework for analyzing convergence in Score-based Generative Models (SGMs) We show that weak log-concavity of the data distribution evolves into log-concavity over time. Our approach circumvents the need for stringent regularity conditions on the score function and its regularity.
  arXiv  Detail & Related papers  (2025-01-04T14:33:27Z)
• Semiparametric conformal prediction [79.6147286161434]
  Risk-sensitive applications require well-calibrated prediction sets over multiple, potentially correlated target variables. We treat the scores as random vectors and aim to construct the prediction set accounting for their joint correlation structure. We report desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional general score-mismatched diffusion samplers. We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions. This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates [5.13323375365494]
  We provide theoretical guarantees for the convergence behaviour of diffusion-based generative models under strongly log-concave data. Our class of functions used for score estimation is made of Lipschitz continuous functions avoiding any Lipschitzness assumption on the score function. This approach yields the best known convergence rate for our sampling algorithm.
  arXiv  Detail & Related papers  (2023-11-22T18:40:45Z)
• KL Convergence Guarantees for Score diffusion models under minimal data assumptions [9.618473763561418]
  A notable challenge persists in the form of a lack of comprehensive quantitative results. This article focuses on score diffusion models with fixed step size stemming from the Ornstein-Uhlenbeck semigroup and its kinetic counterpart.
  arXiv  Detail & Related papers  (2023-08-23T16:31:08Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• Concrete Score Matching: Generalized Score Matching for Discrete Data [109.12439278055213]
  "Concrete score" is a generalization of the (Stein) score for discrete settings. "Concrete Score Matching" is a framework to learn such scores from samples.
  arXiv  Detail & Related papers  (2022-11-02T00:41:37Z)
• FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation [72.19198763459448]
  We learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise. These perturbed data densities are linked together by the Fokker-Planck equation (FPE), a partial differential equation (PDE) governing the spatial-temporal evolution of a density. We derive a corresponding equation called the score FPE that characterizes the noise-conditional scores of the perturbed data densities.
  arXiv  Detail & Related papers  (2022-10-09T16:27:25Z)
• SIXO: Smoothing Inference with Twisted Objectives [8.049531918823758]
  We introduce SIXO, a method that learns targets that approximate the smoothing distributions. We then use SMC with these learned targets to define a variational objective for model and proposal learning.
  arXiv  Detail & Related papers  (2022-06-13T07:46:35Z)
• Generative Modeling with Denoising Auto-Encoders and Langevin Sampling [88.83704353627554]
  We show that both DAE and DSM provide estimates of the score of the smoothed population density. We then apply our results to the homotopy method of arXiv:1907.05600 and provide theoretical justification for its empirical success.
  arXiv  Detail & Related papers  (2020-01-31T23:50:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.