FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation

• URL: http://arxiv.org/abs/2210.04296v4
• Date: Wed, 14 Jun 2023 05:26:28 GMT
• Title: FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation
• Authors: Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, Stefano Ermon
• Abstract summary: We learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise. These perturbed data densities are linked together by the Fokker-Planck equation (FPE), a partial differential equation (PDE) governing the spatial-temporal evolution of a density. We derive a corresponding equation called the score FPE that characterizes the noise-conditional scores of the perturbed data densities.
• Score: 72.19198763459448
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Score-based generative models (SGMs) learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise. These perturbed data densities are linked together by the Fokker-Planck equation (FPE), a partial differential equation (PDE) governing the spatial-temporal evolution of a density undergoing a diffusion process. In this work, we derive a corresponding equation called the score FPE that characterizes the noise-conditional scores of the perturbed data densities (i.e., their gradients). Surprisingly, despite the impressive empirical performance, we observe that scores learned through denoising score matching (DSM) fail to fulfill the underlying score FPE, which is an inherent self-consistency property of the ground truth score. We prove that satisfying the score FPE is desirable as it improves the likelihood and the degree of conservativity. Hence, we propose to regularize the DSM objective to enforce satisfaction of the score FPE, and we show the effectiveness of this approach across various datasets.

Related papers

• DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval [49.076590578101985]
  We present a diffusion-based ATR framework (DiffATR) that generates joint distribution from noise. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-16T06:33:26Z)
• Iso-Diffusion: Improving Diffusion Probabilistic Models Using the Isotropy of the Additive Gaussian Noise [0.0]
  Minimizing the mean squared error between the additive and predicted noise alone does not impose constraints on the predicted noise to be isotropic. We utilize the isotropy of the additive noise as a constraint on the objective function to enhance the fidelity of DDPMs.
  arXiv  Detail & Related papers  (2024-03-25T14:05:52Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Concrete Score Matching: Generalized Score Matching for Discrete Data [109.12439278055213]
  "Concrete score" is a generalization of the (Stein) score for discrete settings. "Concrete Score Matching" is a framework to learn such scores from samples.
  arXiv  Detail & Related papers  (2022-11-02T00:41:37Z)
• Denoising Likelihood Score Matching for Conditional Score-based Data Generation [22.751924447125955]
  We propose a novel training objective called Denoising Likelihood Score Matching (DLSM) loss to match the gradients of the true log likelihood density. Our experimental evidence shows that the proposed method outperforms the previous methods noticeably in terms of several key evaluation metrics.
  arXiv  Detail & Related papers  (2022-03-27T04:37:54Z)
• Estimating High Order Gradients of the Data Distribution by Denoising [81.24581325617552]
  First order derivative of a data density can be estimated efficiently by denoising score matching. We propose a method to directly estimate high order derivatives (scores) of a data density from samples.
  arXiv  Detail & Related papers  (2021-11-08T18:59:23Z)
• Score Matching Model for Unbounded Data Score [23.708122045184695]
  In real datasets, the score function diverges as the perturbation noise ($sigma$) decreases to zero. We introduce Unbounded Noise Score Network (UNCSN) that resolves the score problem. We also introduce a new type of SDE, so the exact log likelihood can be calculated from the newly suggested SDE.
  arXiv  Detail & Related papers  (2021-06-10T06:30:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.