Score diffusion models without early stopping: finite Fisher information
is all you need
- URL: http://arxiv.org/abs/2308.12240v1
- Date: Wed, 23 Aug 2023 16:31:08 GMT
- Title: Score diffusion models without early stopping: finite Fisher information
is all you need
- Authors: Giovanni Conforti, Alain Durmus, Marta Gentiloni Silveri
- Abstract summary: A notable challenge persists in the form of a lack of comprehensive quantitative results.
In almost all reported bounds in Kullback Leibler (KL) divergence, it is assumed that either the score function or its approximation is Lipschitz uniformly in time.
In this article, we tackle the aforementioned limitations by focusing on score diffusion models with fixed step size stemming from the Ornstein-Ulhenbeck semigroup and its kinetic counterpart.
- Score: 10.810200596141332
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Diffusion models are a new class of generative models that revolve around the
estimation of the score function associated with a stochastic differential
equation. Subsequent to its acquisition, the approximated score function is
then harnessed to simulate the corresponding time-reversal process, ultimately
enabling the generation of approximate data samples. Despite their evident
practical significance these models carry, a notable challenge persists in the
form of a lack of comprehensive quantitative results, especially in scenarios
involving non-regular scores and estimators. In almost all reported bounds in
Kullback Leibler (KL) divergence, it is assumed that either the score function
or its approximation is Lipschitz uniformly in time. However, this condition is
very restrictive in practice or appears to be difficult to establish.
To circumvent this issue, previous works mainly focused on establishing
convergence bounds in KL for an early stopped version of the diffusion model
and a smoothed version of the data distribution, or assuming that the data
distribution is supported on a compact manifold. These explorations have lead
to interesting bounds in either Wasserstein or Fortet-Mourier metrics. However,
the question remains about the relevance of such early-stopping procedure or
compactness conditions. In particular, if there exist a natural and mild
condition ensuring explicit and sharp convergence bounds in KL.
In this article, we tackle the aforementioned limitations by focusing on
score diffusion models with fixed step size stemming from the
Ornstein-Ulhenbeck semigroup and its kinetic counterpart. Our study provides a
rigorous analysis, yielding simple, improved and sharp convergence bounds in KL
applicable to any data distribution with finite Fisher information with respect
to the standard Gaussian distribution.
Related papers
- Inference at the data's edge: Gaussian processes for modeling and inference under model-dependency, poor overlap, and extrapolation [0.0]
The Gaussian Process (GP) is a flexible non-linear regression approach.
It provides a principled approach to handling our uncertainty over predicted (counterfactual) values.
This is especially valuable under conditions of extrapolation or weak overlap.
arXiv Detail & Related papers (2024-07-15T05:09:50Z) - Tamed Langevin sampling under weaker conditions [27.872857402255775]
We investigate the problem of sampling from distributions that are not log-concave and are only weakly dissipative.
We introduce a taming scheme which is tailored to the growth and decay properties of the target distribution.
We provide explicit non-asymptotic guarantees for the proposed sampler in terms of the Kullback-Leibler divergence, total variation, and Wasserstein distance to the target distribution.
arXiv Detail & Related papers (2024-05-27T23:00:40Z) - Non-asymptotic Convergence of Discrete-time Diffusion Models: New Approach and Improved Rate [49.97755400231656]
We establish convergence guarantees for substantially larger classes of distributions under DT diffusion processes.
We then specialize our results to a number of interesting classes of distributions with explicit parameter dependencies.
We propose a novel accelerated sampler and show that it improves the convergence rates of the corresponding regular sampler by orders of magnitude with respect to all system parameters.
arXiv Detail & Related papers (2024-02-21T16:11:47Z) - An analysis of the noise schedule for score-based generative models [7.180235086275926]
We establish an upper bound for the KL divergence between the target and the estimated distributions.
We provide a tighter error bound in Wasserstein distance compared to state-of-the-art results.
arXiv Detail & Related papers (2024-02-07T08:24:35Z) - A Note on the Convergence of Denoising Diffusion Probabilistic Models [3.75292409381511]
We derive a quantitative upper bound on the Wasserstein distance between the data-generating distribution and the distribution learned by a diffusion model.
Unlike previous works in this field, our result does not make assumptions on the learned score function.
arXiv Detail & Related papers (2023-12-10T20:29:58Z) - Eliminating Lipschitz Singularities in Diffusion Models [51.806899946775076]
We show that diffusion models frequently exhibit the infinite Lipschitz near the zero point of timesteps.
This poses a threat to the stability and accuracy of the diffusion process, which relies on integral operations.
We propose a novel approach, dubbed E-TSDM, which eliminates the Lipschitz of the diffusion model near zero.
arXiv Detail & Related papers (2023-06-20T03:05:28Z) - ChiroDiff: Modelling chirographic data with Diffusion Models [132.5223191478268]
We introduce a powerful model-class namely "Denoising Diffusion Probabilistic Models" or DDPMs for chirographic data.
Our model named "ChiroDiff", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher temporal sampling rate.
arXiv Detail & Related papers (2023-04-07T15:17:48Z) - Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain.
We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions.
We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
arXiv Detail & Related papers (2022-11-30T05:33:29Z) - Convergence of denoising diffusion models under the manifold hypothesis [3.096615629099617]
Denoising diffusion models are a recent class of generative models exhibiting state-of-the-art performance in image and audio synthesis.
This paper provides the first convergence results for diffusion models in a more general setting.
arXiv Detail & Related papers (2022-08-10T12:50:47Z) - Posterior Coreset Construction with Kernelized Stein Discrepancy for
Model-Based Reinforcement Learning [78.30395044401321]
We develop a novel model-based approach to reinforcement learning (MBRL)
It relaxes the assumptions on the target transition model to belong to a generic family of mixture models.
It can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.
arXiv Detail & Related papers (2022-06-02T17:27:49Z) - Efficient CDF Approximations for Normalizing Flows [64.60846767084877]
We build upon the diffeomorphic properties of normalizing flows to estimate the cumulative distribution function (CDF) over a closed region.
Our experiments on popular flow architectures and UCI datasets show a marked improvement in sample efficiency as compared to traditional estimators.
arXiv Detail & Related papers (2022-02-23T06:11:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.