Lecture Notes in Probabilistic Diffusion Models
- URL: http://arxiv.org/abs/2312.10393v1
- Date: Sat, 16 Dec 2023 09:36:54 GMT
- Title: Lecture Notes in Probabilistic Diffusion Models
- Authors: Inga Str\"umke, Helge Langseth
- Abstract summary: Diffusion models are loosely modelled based on non-equilibrium thermodynamics.
The diffusion model learns the data manifold to which the original and thus the reconstructed data samples belong.
Diffusion models have -- unlike variational autoencoder and flow models -- latent variables with the same dimensionality as the original data.
- Score: 0.5361320134021585
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Diffusion models are loosely modelled based on non-equilibrium
thermodynamics, where \textit{diffusion} refers to particles flowing from
high-concentration regions towards low-concentration regions. In statistics,
the meaning is quite similar, namely the process of transforming a complex
distribution $p_{\text{complex}}$ on $\mathbb{R}^d$ to a simple distribution
$p_{\text{prior}}$ on the same domain. This constitutes a Markov chain of
diffusion steps of slowly adding random noise to data, followed by a reverse
diffusion process in which the data is reconstructed from the noise. The
diffusion model learns the data manifold to which the original and thus the
reconstructed data samples belong, by training on a large number of data
points. While the diffusion process pushes a data sample off the data manifold,
the reverse process finds a trajectory back to the data manifold. Diffusion
models have -- unlike variational autoencoder and flow models -- latent
variables with the same dimensionality as the original data, and they are
currently\footnote{At the time of writing, 2023.} outperforming other
approaches -- including Generative Adversarial Networks (GANs) -- to modelling
the distribution of, e.g., natural images.
Related papers
- Continuous Diffusion Model for Language Modeling [57.396578974401734]
Existing continuous diffusion models for discrete data have limited performance compared to discrete approaches.
We propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution.
arXiv Detail & Related papers (2025-02-17T08:54:29Z) - Graph Representation Learning with Diffusion Generative Models [0.0]
We train a discrete diffusion model within an autoencoder framework to learn meaningful embeddings for graph data.
Our approach demonstrates the potential of discrete diffusion models to be used for graph representation learning.
arXiv Detail & Related papers (2025-01-22T07:12:10Z) - Non-Normal Diffusion Models [3.5534933448684134]
Diffusion models generate samples by incrementally reversing a process that turns data into noise.
We show that when the step size goes to zero, the reversed process is invariant to the distribution of these increments.
We demonstrate the effectiveness of these models on density estimation and generative modeling tasks on standard image datasets.
arXiv Detail & Related papers (2024-12-10T21:31:12Z) - Constrained Diffusion Models via Dual Training [80.03953599062365]
Diffusion processes are prone to generating samples that reflect biases in a training dataset.
We develop constrained diffusion models by imposing diffusion constraints based on desired distributions.
We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
arXiv Detail & Related papers (2024-08-27T14:25:42Z) - Renormalizing Diffusion Models [0.7252027234425334]
We use diffusion models to learn inverse renormalization group flows of statistical and quantum field theories.
Our work provides an interpretation of multiscale diffusion models, and gives physically-inspired suggestions for diffusion models which should have novel properties.
arXiv Detail & Related papers (2023-08-23T18:02:31Z) - Score Approximation, Estimation and Distribution Recovery of Diffusion
Models on Low-Dimensional Data [68.62134204367668]
This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace.
We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated.
The generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution.
arXiv Detail & Related papers (2023-02-14T17:02:35Z) - Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain.
We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions.
We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
arXiv Detail & Related papers (2022-11-30T05:33:29Z) - From Points to Functions: Infinite-dimensional Representations in
Diffusion Models [23.916417852496608]
Diffusion-based generative models learn to iteratively transfer unstructured noise to a complex target distribution.
We show that a combination of information content from different time steps gives a strictly better representation for the downstream task.
arXiv Detail & Related papers (2022-10-25T05:30:53Z) - Diffusion Models in Vision: A Survey [73.10116197883303]
A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage.
Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens.
arXiv Detail & Related papers (2022-09-10T22:00:30Z) - Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial
Auto-Encoders [137.1060633388405]
Diffusion-based generative models learn how to generate the data by inferring a reverse diffusion chain.
We propose a faster and cheaper approach that adds noise not until the data become pure random noise.
We show that the proposed model can be cast as an adversarial auto-encoder empowered by both the diffusion process and a learnable implicit prior.
arXiv Detail & Related papers (2022-02-19T20:18:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.