Interpreting diffusion score matching using normalizing flow

• URL: http://arxiv.org/abs/2107.10072v1
• Date: Wed, 21 Jul 2021 13:27:32 GMT
• Title: Interpreting diffusion score matching using normalizing flow
• Abstract summary: We show that diffusion score matching (DSM) (or diffusion Stein discrepancy (DSD)) is equivalent to the original score matching (or Stein discrepancy) evaluated in a normalizing flow. Specifically, we prove that DSM (or DSD) is equivalent to the original score matching (or Stein discrepancy) evaluated in the transformed space defined by the normalizing flow.
• Score: 22.667661526643265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scoring matching (SM), and its related counterpart, Stein discrepancy (SD) have achieved great success in model training and evaluations. However, recent research shows their limitations when dealing with certain types of distributions. One possible fix is incorporating the original score matching (or Stein discrepancy) with a diffusion matrix, which is called diffusion score matching (DSM) (or diffusion Stein discrepancy (DSD)). However, the lack of interpretation of the diffusion limits its usage within simple distributions and manually chosen matrix. In this work, we plan to fill this gap by interpreting the diffusion matrix using normalizing flows. Specifically, we theoretically prove that DSM (or DSD) is equivalent to the original score matching (or Stein discrepancy) evaluated in the transformed space defined by the normalizing flow, where the diffusion matrix is the inverse of the flow's Jacobian matrix. In addition, we also build its connection to Riemannian manifolds and further extend it to continuous flows, where the change of DSM is characterized by an ODE.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• Measuring and Decomposing Mode Separation via the Canonical Diffusion [0.0]
  Mode separation is a fundamental geometric property of densities, difficult to quantify in high dimensions.<n>We measure mode separation through a single process intrinsic to the density.<n>We apply our framework to three settings: (i) synthetic Gaussian mixtures, where SSA tracks mutual information; (ii) SDXL text-to-image generations, where SSA and DA capture structure that entropy and PCA miss; and (iii) molecular dynamics of alanine dipeptide, where DA recovers the known slow backbones from static samples alone.
  arXiv  Detail & Related papers  (2026-05-09T08:05:55Z)
• A Unified View of Drifting and Score-Based Models [92.51878018325031]
  We make our relationship to the score-matching principle behind diffusion models precise by showing that drifting admits a score-based formulation on kernel-smoothed distributions.<n>We derive an exact decomposition for general radial kernels, and for the Laplace kernel we prove rigorous error bounds showing that drifting remains an accurate proxy for score matching in low-temperature and high-dimensional regimes.
  arXiv  Detail & Related papers  (2026-03-08T07:41:36Z)
• Diffusion Models: A Mathematical Introduction [3.8673630752805437]
  We present a self-contained derivation of diffusion-based generative models.<n>We construct denoising diffusion probabilistic models from first principles.<n>Readers can both follow the theory and implement the corresponding algorithms in practice.
  arXiv  Detail & Related papers  (2025-11-13T16:20:52Z)
• Authentic Discrete Diffusion Model [72.31371542619121]
  Authentic Discrete Diffusion (ADD) framework redefines prior pseudo-discrete approaches.<n>ADD reformulates the diffusion input by directly using float-encoded one-hot class data.<n> experiments demonstrate that ADD achieves superior performance on classification tasks compared to the baseline.
  arXiv  Detail & Related papers  (2025-10-01T15:51:10Z)
• EquiReg: Equivariance Regularized Diffusion for Inverse Problems [67.01847869495558]
  We propose EquiReg diffusion, a framework for regularizing posterior sampling in diffusion-based inverse problem solvers.<n>When applied to a variety of solvers, EquiReg outperforms state-of-the-art diffusion models in both linear and nonlinear image restoration tasks.
  arXiv  Detail & Related papers  (2025-05-29T01:25:43Z)
• Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Study [3.265950484493743]
  Diffusion models can be prone to memorization.<n>Regularization on the score has the same effect as increasing the size of the training dataset.<n>This perspective highlights two regularization mechanisms taking place in denoising diffusions.
  arXiv  Detail & Related papers  (2025-05-28T20:22:18Z)
• A Mixture-Based Framework for Guiding Diffusion Models [19.83064246586143]
  Denoising diffusion models have driven significant progress in the field of Bayesian inverse problems. Recent approaches use pre-trained diffusion models as priors to solve a wide range of such problems. This work proposes a novel mixture approximation of these intermediate distributions.
  arXiv  Detail & Related papers  (2025-02-05T16:26:06Z)
• Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models [22.39558434131574]
  Existing data attribution methods for diffusion models typically quantify the contribution of a training sample. We argue that the direct usage of diffusion loss cannot represent such a contribution accurately due to the calculation of diffusion loss. We propose Diffusion Attribution Score (textitDAS) to measure the direct comparison between predicted distributions with an attribution score.
  arXiv  Detail & Related papers  (2024-10-24T10:58:17Z)
• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional general score-mismatched diffusion samplers. We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions. This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• Solving Prior Distribution Mismatch in Diffusion Models via Optimal Transport [24.90486913773359]
  In recent years, the knowledge surrounding diffusion models(DMs) has grown significantly, though several theoretical gaps remain. This paper explores the deeper relationship between optimal transport(OT) theory and DMs with discrete initial distribution. We prove that as the diffusion termination time increases, the probability flow exponentially converges to the gradient of the solution to the classical Monge-Ampere equation.
  arXiv  Detail & Related papers  (2024-10-17T10:54:55Z)
• G2D2: Gradient-Guided Discrete Diffusion for Inverse Problem Solving [83.56510119503267]
  This paper presents a novel method for addressing linear inverse problems by leveraging generative models based on discrete diffusion as priors.<n>We employ a star-shaped noise process to mitigate the drawbacks of traditional discrete diffusion models with absorbing states.
  arXiv  Detail & Related papers  (2024-10-09T06:18:25Z)
• Amortizing intractable inference in diffusion models for vision, language, and control [89.65631572949702]
  This paper studies amortized sampling of the posterior over data, $mathbfxsim prm post(mathbfx)propto p(mathbfx)r(mathbfx)$, in a model that consists of a diffusion generative model prior $p(mathbfx)$ and a black-box constraint or function $r(mathbfx)$. We prove the correctness of a data-free learning objective, relative trajectory balance, for training a diffusion model that samples from
  arXiv  Detail & Related papers  (2024-05-31T16:18:46Z)
• Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective [18.331374727331077]
  We provide a theoretical understanding of the Lipschitz continuity and second momentum properties of the diffusion process. Our results provide deeper theoretical insights into the dynamics of the diffusion process under common data distributions.
  arXiv  Detail & Related papers  (2024-05-26T03:32:27Z)
• Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models [59.331993845831946]
  Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. This paper provides the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models.
  arXiv  Detail & Related papers  (2024-03-03T23:15:48Z)
• Eliminating Lipschitz Singularities in Diffusion Models [51.806899946775076]
  We show that diffusion models frequently exhibit the infinite Lipschitz near the zero point of timesteps. This poses a threat to the stability and accuracy of the diffusion process, which relies on integral operations. We propose a novel approach, dubbed E-TSDM, which eliminates the Lipschitz of the diffusion model near zero.
  arXiv  Detail & Related papers  (2023-06-20T03:05:28Z)
• Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems [64.29491112653905]
  We propose a novel and efficient diffusion sampling strategy that synergistically combines the diffusion sampling and Krylov subspace methods. Specifically, we prove that if tangent space at a denoised sample by Tweedie's formula forms a Krylov subspace, then the CG with the denoised data ensures the data consistency update to remain in the tangent space. Our proposed method achieves more than 80 times faster inference time than the previous state-of-the-art method.
  arXiv  Detail & Related papers  (2023-03-10T07:42:49Z)
• Denoising Diffusion Samplers [41.796349001299156]
  Denoising diffusion models are a popular class of generative models providing state-of-the-art results in many domains. We explore a similar idea to sample approximately from unnormalized probability density functions and estimate their normalizing constants. While score matching is not applicable in this context, we can leverage many of the ideas introduced in generative modeling for Monte Carlo sampling.
  arXiv  Detail & Related papers  (2023-02-27T14:37:16Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Efficient CDF Approximations for Normalizing Flows [64.60846767084877]
  We build upon the diffeomorphic properties of normalizing flows to estimate the cumulative distribution function (CDF) over a closed region. Our experiments on popular flow architectures and UCI datasets show a marked improvement in sample efficiency as compared to traditional estimators.
  arXiv  Detail & Related papers  (2022-02-23T06:11:49Z)
• Resampling Base Distributions of Normalizing Flows [0.0]
  We introduce a base distribution for normalizing flows based on learned rejection sampling. We develop suitable learning algorithms using both maximizing the log-likelihood and the optimization of the reverse Kullback-Leibler divergence.
  arXiv  Detail & Related papers  (2021-10-29T14:44:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.