FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation

• URL: http://arxiv.org/abs/2210.04296v4
• Date: Wed, 14 Jun 2023 05:26:28 GMT
• Title: FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation
• Authors: Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, Stefano Ermon
• Abstract summary: We learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise. These perturbed data densities are linked together by the Fokker-Planck equation (FPE), a partial differential equation (PDE) governing the spatial-temporal evolution of a density. We derive a corresponding equation called the score FPE that characterizes the noise-conditional scores of the perturbed data densities.
• Score: 72.19198763459448
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Score-based generative models (SGMs) learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise. These perturbed data densities are linked together by the Fokker-Planck equation (FPE), a partial differential equation (PDE) governing the spatial-temporal evolution of a density undergoing a diffusion process. In this work, we derive a corresponding equation called the score FPE that characterizes the noise-conditional scores of the perturbed data densities (i.e., their gradients). Surprisingly, despite the impressive empirical performance, we observe that scores learned through denoising score matching (DSM) fail to fulfill the underlying score FPE, which is an inherent self-consistency property of the ground truth score. We prove that satisfying the score FPE is desirable as it improves the likelihood and the degree of conservativity. Hence, we propose to regularize the DSM objective to enforce satisfaction of the score FPE, and we show the effectiveness of this approach across various datasets.

Related papers

• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models [11.743167854433306]
  Diffusion models generate samples by estimating the score function of the target distribution at various noise levels. In this work, we establish the first (nearly) dimension-free sample bounds complexity for learning these score functions. A key aspect of our analysis is the use of a single function approximator to jointly estimate scores across noise levels.
  arXiv  Detail & Related papers  (2025-02-14T18:32:22Z)
• DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval [49.076590578101985]
  We present a diffusion-based ATR framework (DiffATR) that generates joint distribution from noise. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-16T06:33:26Z)
• Iso-Diffusion: Improving Diffusion Probabilistic Models Using the Isotropy of the Additive Gaussian Noise [0.0]
  We show how to use the isotropy of the additive noise as a constraint on the objective function to enhance the fidelity of DDPMs. Our approach is simple and can be applied to any DDPM variant.
  arXiv  Detail & Related papers  (2024-03-25T14:05:52Z)
• An analysis of the noise schedule for score-based generative models [7.180235086275926]
  Score-based generative models (SGMs) aim at estimating a target data distribution by learning score functions using only noise-perturbed samples from the target. Recent literature has focused extensively on assessing the error between the target and estimated distributions, gauging the generative quality through the Kullback-Leibler (KL) divergence and Wasserstein distances. We establish an upper bound for the KL divergence between the target and the estimated distributions, explicitly depending on any time-dependent noise schedule.
  arXiv  Detail & Related papers  (2024-02-07T08:24:35Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Concrete Score Matching: Generalized Score Matching for Discrete Data [109.12439278055213]
  "Concrete score" is a generalization of the (Stein) score for discrete settings. "Concrete Score Matching" is a framework to learn such scores from samples.
  arXiv  Detail & Related papers  (2022-11-02T00:41:37Z)
• Denoising Likelihood Score Matching for Conditional Score-based Data Generation [22.751924447125955]
  We propose a novel training objective called Denoising Likelihood Score Matching (DLSM) loss to match the gradients of the true log likelihood density. Our experimental evidence shows that the proposed method outperforms the previous methods noticeably in terms of several key evaluation metrics.
  arXiv  Detail & Related papers  (2022-03-27T04:37:54Z)
• Estimating High Order Gradients of the Data Distribution by Denoising [81.24581325617552]
  First order derivative of a data density can be estimated efficiently by denoising score matching. We propose a method to directly estimate high order derivatives (scores) of a data density from samples.
  arXiv  Detail & Related papers  (2021-11-08T18:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.