Score-Based Generative Models Detect Manifolds

• URL: http://arxiv.org/abs/2206.01018v1
• Date: Thu, 2 Jun 2022 12:29:10 GMT
• Title: Score-Based Generative Models Detect Manifolds
• Authors: Jakiw Pidstrigach
• Abstract summary: Score-based generative models (SGMs) need to approximate the scores $nabla log p_t$ of the intermediate distributions as well as the final distribution $p_T$ of the forward process. We find precise conditions under which SGMs are able to produce samples from an underlying (low-dimensional) data manifold.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Score-based generative models (SGMs) need to approximate the scores $\nabla \log p_t$ of the intermediate distributions as well as the final distribution $p_T$ of the forward process. The theoretical underpinnings of the effects of these approximations are still lacking. We find precise conditions under which SGMs are able to produce samples from an underlying (low-dimensional) data manifold $\mathcal{M}$. This assures us that SGMs are able to generate the "right kind of samples". For example, taking $\mathcal{M}$ to be the subset of images of faces, we find conditions under which the SGM robustly produces an image of a face, even though the relative frequencies of these images might not accurately represent the true data generating distribution. Moreover, this analysis is a first step towards understanding the generalization properties of SGMs: Taking $\mathcal{M}$ to be the set of all training samples, our results provide a precise description of when the SGM memorizes its training data.

Related papers

• Going beyond Compositions, DDPMs Can Produce Zero-Shot Interpolations [54.95457207525101]
  Denoising Diffusion Probabilistic Models (DDPMs) exhibit remarkable capabilities in image generation. We study DDPMs trained on strictly separate subsets of the data distribution with large gaps on the support of latent factors. We show that such a model can effectively generate images in the unexplored, intermediate regions of the distribution.
  arXiv  Detail & Related papers  (2024-05-29T15:41:53Z)
• A Good Score Does not Lead to A Good Generative Model [14.752242187781107]
  Score-based Generative Models (SGMs) is one leading method in generative modeling. We show that SGMs can generate samples from a distribution that is close to the ground-truth if the underlying score function is learned well.
  arXiv  Detail & Related papers  (2024-01-10T00:17:36Z)
• Preconditioned Score-based Generative Models [49.88840603798831]
  An intuitive acceleration method is to reduce the sampling iterations which however causes severe performance degradation. We propose a model-agnostic bfem preconditioned diffusion sampling (PDS) method that leverages matrix preconditioning to alleviate the aforementioned problem. PDS alters the sampling process of a vanilla SGM at marginal extra computation cost, and without model retraining.
  arXiv  Detail & Related papers  (2023-02-13T16:30:53Z)
• Provably Reliable Large-Scale Sampling from Gaussian Processes [3.6417668958891793]
  We demonstrate how to generate a size (n) sample from approximate Gaussian process (GP) models. We provide guarantees that, with high probability, the sample is indistinguishable from a sample from the desired GP.
  arXiv  Detail & Related papers  (2022-11-15T10:36:21Z)
• Convergence for score-based generative modeling with polynomial complexity [9.953088581242845]
  We prove the first convergence guarantees for the core mechanic behind Score-based generative modeling. Compared to previous works, we do not incur error that grows exponentially in time or that suffers from a curse of dimensionality. We show that a predictor-corrector gives better convergence than using either portion alone.
  arXiv  Detail & Related papers  (2022-06-13T14:57:35Z)
• Riemannian Score-Based Generative Modeling [56.20669989459281]
  We introduce score-based generative models (SGMs) demonstrating remarkable empirical performance. Current SGMs make the underlying assumption that the data is supported on a Euclidean manifold with flat geometry. This prevents the use of these models for applications in robotics, geoscience or protein modeling.
  arXiv  Detail & Related papers  (2022-02-06T11:57:39Z)
• Reversible Gromov-Monge Sampler for Simulation-Based Inference [3.725559762520257]
  This paper introduces a new simulation-based inference procedure to model and sample from multi-dimensional probability distributions. Motivated by the work of M'emoli (2011) and Sturm (2012) on distance and isomorphism between metric measure spaces, we propose a new notion called the Reversible Gromov-Monge (RGM) distance.
  arXiv  Detail & Related papers  (2021-09-28T23:09:24Z)
• Score-based Generative Modeling in Latent Space [93.8985523558869]
  Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space.
  arXiv  Detail & Related papers  (2021-06-10T17:26:35Z)
• Continual Learning with Fully Probabilistic Models [70.3497683558609]
  We present an approach for continual learning based on fully probabilistic (or generative) models of machine learning. We propose a pseudo-rehearsal approach using a Gaussian Mixture Model (GMM) instance for both generator and classifier functionalities. We show that GMR achieves state-of-the-art performance on common class-incremental learning problems at very competitive time and memory complexity.
  arXiv  Detail & Related papers  (2021-04-19T12:26:26Z)
• Adversarially-learned Inference via an Ensemble of Discrete Undirected Graphical Models [3.04585143845864]
  We propose an inference-agnostic adversarial training framework which produces an infinitely-large ensemble of graphical models (AGMs) AGMs show significantly better generalization to unseen inference tasks compared to EGMs, as well as deep neural architectures like GibbsNet and VAEAC.
  arXiv  Detail & Related papers  (2020-07-09T19:13:36Z)
• Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model [50.38446482252857]
  This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator) We first consider $gamma$-discounted infinite-horizon Markov decision processes (MDPs) with state space $mathcalS$ and action space $mathcalA$. We prove that a plain model-based planning algorithm suffices to achieve minimax-optimal sample complexity given any target accuracy level.
  arXiv  Detail & Related papers  (2020-05-26T17:53:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.