Eliminating Lipschitz Singularities in Diffusion Models

• URL: http://arxiv.org/abs/2306.11251v1
• Date: Tue, 20 Jun 2023 03:05:28 GMT
• Title: Eliminating Lipschitz Singularities in Diffusion Models
• Authors: Zhantao Yang, Ruili Feng, Han Zhang, Yujun Shen, Kai Zhu, Lianghua Huang, Yifei Zhang, Yu Liu, Deli Zhao, Jingren Zhou, Fan Cheng
• Abstract summary: We show that diffusion models frequently exhibit the infinite Lipschitz near the zero point of timesteps. This poses a threat to the stability and accuracy of the diffusion process, which relies on integral operations. We propose a novel approach, dubbed E-TSDM, which eliminates the Lipschitz of the diffusion model near zero.
• Score: 51.806899946775076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models. However, the rationality of the diffusion process itself receives limited attention, leaving the question of whether the problem is well-posed and well-conditioned. In this paper, we uncover a vexing propensity of diffusion models: they frequently exhibit the infinite Lipschitz near the zero point of timesteps. This poses a threat to the stability and accuracy of the diffusion process, which relies on integral operations. We provide a comprehensive evaluation of the issue from both theoretical and empirical perspectives. To address this challenge, we propose a novel approach, dubbed E-TSDM, which eliminates the Lipschitz singularity of the diffusion model near zero. Remarkably, our technique yields a substantial improvement in performance, e.g., on the high-resolution FFHQ dataset ($256\times256$). Moreover, as a byproduct of our method, we manage to achieve a dramatic reduction in the Frechet Inception Distance of other acceleration methods relying on network Lipschitz, including DDIM and DPM-Solver, by over 33$\%$. We conduct extensive experiments on diverse datasets to validate our theory and method. Our work not only advances the understanding of the general diffusion process, but also provides insights for the design of diffusion models.

Related papers

• Distillation of Discrete Diffusion through Dimensional Correlations [21.078500510691747]
  "Mixture" models in discrete diffusion are capable of treating dimensional correlations while remaining scalable. We empirically demonstrate that our proposed method for discrete diffusions work in practice, by distilling a continuous-time discrete diffusion model pretrained on the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2024-10-11T10:53:03Z)
• Variational Distillation of Diffusion Policies into Mixture of Experts [26.315682445979302]
  This work introduces Variational Diffusion Distillation (VDD), a novel method that distills denoising diffusion policies into Mixtures of Experts (MoE) Diffusion Models are the current state-of-the-art in generative modeling due to their exceptional ability to accurately learn and represent complex, multi-modal distributions. VDD is the first method that distills pre-trained diffusion models into MoE models, and hence, combines the expressiveness of Diffusion Models with the benefits of Mixture Models.
  arXiv  Detail & Related papers  (2024-06-18T12:15:05Z)
• Intention-aware Denoising Diffusion Model for Trajectory Prediction [14.524496560759555]
  Trajectory prediction is an essential component in autonomous driving, particularly for collision avoidance systems. We propose utilizing the diffusion model to generate the distribution of future trajectories. We propose an Intention-aware denoising Diffusion Model (IDM) Our methods achieve state-of-the-art results, with an FDE of 13.83 pixels on the SDD dataset and 0.36 meters on the ETH/UCY dataset.
  arXiv  Detail & Related papers  (2024-03-14T09:05:25Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models [76.46246743508651]
  We show that current diffusion models actually have an expressive bottleneck in backward denoising. We introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising.
  arXiv  Detail & Related papers  (2023-09-25T12:03:32Z)
• Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations [6.2284442126065525]
  We study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient.
  arXiv  Detail & Related papers  (2023-06-03T09:27:15Z)
• Reconstructing Graph Diffusion History from a Single Snapshot [87.20550495678907]
  We propose a novel barycenter formulation for reconstructing Diffusion history from A single SnapsHot (DASH) We prove that estimation error of diffusion parameters is unavoidable due to NP-hardness of diffusion parameter estimation. We also develop an effective solver named DIffusion hiTting Times with Optimal proposal (DITTO)
  arXiv  Detail & Related papers  (2023-06-01T09:39:32Z)
• Diffusion Models are Minimax Optimal Distribution Estimators [49.47503258639454]
  We provide the first rigorous analysis on approximation and generalization abilities of diffusion modeling. We show that when the true density function belongs to the Besov space and the empirical score matching loss is properly minimized, the generated data distribution achieves the nearly minimax optimal estimation rates.
  arXiv  Detail & Related papers  (2023-03-03T11:31:55Z)
• Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions [22.04182099405728]
  Diffusion-based generative models (DBGMs) perturb data to a target noise distribution and reverse this inference diffusion process to generate samples. We show how to maximize a lower-bound on the likelihood for any number of auxiliary variables. We then demonstrate how to parameterize the diffusion for a specified target noise distribution.
  arXiv  Detail & Related papers  (2023-02-14T18:57:04Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.