Linear Convergence of Diffusion Models Under the Manifold Hypothesis

• URL: http://arxiv.org/abs/2410.09046v1
• Date: Fri, 11 Oct 2024 17:58:30 GMT
• Title: Linear Convergence of Diffusion Models Under the Manifold Hypothesis
• Authors: Peter Potaptchik, Iskander Azangulov, George Deligiannidis,
• Abstract summary: We show that the number of steps to converge in Kullback-Leibler(KL) divergence is linear (up to logarithmic terms) in the intrinsic dimension $Leid$. We also show that this linear dependency is sharp.
• Score: 5.040884755454258
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Score-matching generative models have proven successful at sampling from complex high-dimensional data distributions. In many applications, this distribution is believed to concentrate on a much lower $d$-dimensional manifold embedded into $D$-dimensional space; this is known as the manifold hypothesis. The current best-known convergence guarantees are either linear in $D$ or polynomial (superlinear) in $d$. The latter exploits a novel integration scheme for the backward SDE. We take the best of both worlds and show that the number of steps diffusion models require in order to converge in Kullback-Leibler~(KL) divergence is linear (up to logarithmic terms) in the intrinsic dimension $d$. Moreover, we show that this linear dependency is sharp.

Related papers

• Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions [6.9408143976091745]
  Denoising Diffusion Probabilistic Models (DDPM) are powerful state-of-the-art methods used to generate synthetic data from high-dimensional data distributions. We study DDPMs under the manifold hypothesis and prove that they achieve rates independent of the ambient dimension in terms of learning the score. In terms of sampling, we obtain rates independent of the ambient dimension w.r.t. the Kullback-Leibler divergence, and $O(sqrtD)$ w.r.t. the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-09-27T14:57:18Z)
• Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity [11.71206628091551]
  Diffusion models are costly to train and evaluate, reducing the inference cost for diffusion models remains a major goal. Inspired by the recent empirical success in accelerating diffusion models via the parallel sampling techniqueciteshih2024parallel, we propose to divide the sampling process into $mathcalO(1)$ blocks with parallelizable Picard iterations within each block. Our results shed light on the potential of fast and efficient sampling of high-dimensional data on fast-evolving modern large-memory GPU clusters.
  arXiv  Detail & Related papers  (2024-05-24T23:59:41Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Scaling Riemannian Diffusion Models [68.52820280448991]
  We show that our method enables us to scale to high dimensional tasks on nontrivial manifold. We model QCD densities on $SU(n)$ lattices and contrastively learned embeddings on high dimensional hyperspheres.
  arXiv  Detail & Related papers  (2023-10-30T21:27:53Z)
• Conformal inference for regression on Riemannian Manifolds [49.7719149179179]
  We investigate prediction sets for regression scenarios when the response variable, denoted by $Y$, resides in a manifold, and the covariable, denoted by X, lies in Euclidean space. We prove the almost sure convergence of the empirical version of these regions on the manifold to their population counterparts.
  arXiv  Detail & Related papers  (2023-10-12T10:56:25Z)
• Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes [57.396578974401734]
  We introduce a principled framework for building a generative diffusion process on general manifold. Instead of following the denoising approach of previous diffusion models, we construct a diffusion process using a mixture of bridge processes. We develop a geometric understanding of the mixture process, deriving the drift as a weighted mean of tangent directions to the data points.
  arXiv  Detail & Related papers  (2023-10-11T06:04:40Z)
• Nearly $d$-Linear Convergence Bounds for Diffusion Models via Stochastic Localization [40.808942894229325]
  We provide the first convergence bounds which are linear in the data dimension. We show that diffusion models require at most $tilde O(fracd log2(1/delta)varepsilon2)$ steps to approximate an arbitrary distribution.
  arXiv  Detail & Related papers  (2023-08-07T16:01:14Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models [49.81937966106691]
  We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
  arXiv  Detail & Related papers  (2023-06-15T16:30:08Z)
• Diagnosing and Fixing Manifold Overfitting in Deep Generative Models [11.82509693248749]
  Likelihood-based, or explicit, deep generative models use neural networks to construct flexible high-dimensional densities. We show that observed data lies on a low-dimensional manifold embedded in high-dimensional ambient space. We propose a class of two-step procedures consisting of a dimensionality reduction step followed by maximum-likelihood density estimation.
  arXiv  Detail & Related papers  (2022-04-14T18:00:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.