Score-based Diffusion Models in Function Space

• URL: http://arxiv.org/abs/2302.07400v2
• Date: Wed, 22 Nov 2023 19:17:08 GMT
• Title: Score-based Diffusion Models in Function Space
• Authors: Jae Hyun Lim, Nikola B. Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, Christopher Pal, Arash Vahdat, Anima Anandkumar
• Abstract summary: Diffusion models have recently emerged as a powerful framework for generative modeling. We introduce a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution.
• Score: 140.792362459734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have recently emerged as a powerful framework for generative modeling. They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising. Despite their tremendous success, they are mostly formulated on finite-dimensional spaces, e.g. Euclidean, limiting their applications to many domains where the data has a functional form such as in scientific computing and 3D geometric data analysis. In this work, we introduce a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. In DDOs, the forward process perturbs input functions gradually using a Gaussian process. The generative process is formulated by integrating a function-valued Langevin dynamic. Our approach requires an appropriate notion of the score for the perturbed data distribution, which we obtain by generalizing denoising score matching to function spaces that can be infinite-dimensional. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost that is independent of the data resolution. We theoretically and numerically verify the applicability of our approach on a set of problems, including generating solutions to the Navier-Stokes equation viewed as the push-forward distribution of forcings from a Gaussian Random Field (GRF).

Related papers

• Accelerated zero-order SGD under high-order smoothness and overparameterized regime [79.85163929026146]
  We present a novel gradient-free algorithm to solve convex optimization problems. Such problems are encountered in medicine, physics, and machine learning. We provide convergence guarantees for the proposed algorithm under both types of noise.
  arXiv  Detail & Related papers  (2024-11-21T10:26:17Z)
• Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors [0.0]
  Diffusion or score-based models recently showed high performance in image generation. We study theoretically the behavior of diffusion models and their numerical implementation when the data distribution is Gaussian.
  arXiv  Detail & Related papers  (2024-05-23T07:28:56Z)
• Weak Collocation Regression for Inferring Stochastic Dynamics with L\'{e}vy Noise [8.15076267771005]
  We propose a weak form of the Fokker-Planck (FP) equation for extracting dynamics with L'evy noise. Our approach can simultaneously distinguish mixed noise types, even in multi-dimensional problems.
  arXiv  Detail & Related papers  (2024-03-13T06:54:38Z)
• Learning Unnormalized Statistical Models via Compositional Optimization [73.30514599338407]
  Noise-contrastive estimation(NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise. In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models.
  arXiv  Detail & Related papers  (2023-06-13T01:18:16Z)
• Learning to Bound Counterfactual Inference in Structural Causal Models from Observational and Randomised Data [64.96984404868411]
  We derive a likelihood characterisation for the overall data that leads us to extend a previous EM-based algorithm. The new algorithm learns to approximate the (unidentifiability) region of model parameters from such mixed data sources. It delivers interval approximations to counterfactual results, which collapse to points in the identifiable case.
  arXiv  Detail & Related papers  (2022-12-06T12:42:11Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Probabilistic Registration for Gaussian Process 3D shape modelling in the presence of extensive missing data [63.8376359764052]
  We propose a shape fitting/registration method based on a Gaussian Processes formulation, suitable for shapes with extensive regions of missing data. Experiments are conducted both for a 2D small dataset with diverse transformations and a 3D dataset of ears.
  arXiv  Detail & Related papers  (2022-03-26T16:48:27Z)
• Inference and De-Noising of Non-Gaussian Particle Distribution Functions: A Generative Modeling Approach [0.0]
  Inference on data produced by numerical simulations generally consists of binning the data to recover the particle distribution function. Here we demonstrate the use of normalizing flows to learn a smooth, tractable approximation to the noisy particle distribution function.
  arXiv  Detail & Related papers  (2021-10-05T16:38:04Z)
• Weak SINDy For Partial Differential Equations [0.0]
  We extend our Weak SINDy (WSINDy) framework to the setting of partial differential equations (PDEs) The elimination of pointwise derivative approximations via the weak form enables effective machine-precision recovery of model coefficients from noise-free data. We demonstrate WSINDy's robustness, speed and accuracy on several challenging PDEs.
  arXiv  Detail & Related papers  (2020-07-06T16:03:51Z)
• The Random Feature Model for Input-Output Maps between Banach Spaces [6.282068591820945]
  The random feature model is a parametric approximation to kernel or regression methods. We propose a methodology for use of the random feature model as a data-driven surrogate for operators that map an input Banach space to an output Banach space.
  arXiv  Detail & Related papers  (2020-05-20T17:41:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.