Score-based Generative Models with Adaptive Momentum

• URL: http://arxiv.org/abs/2405.13726v1
• Date: Wed, 22 May 2024 15:20:27 GMT
• Title: Score-based Generative Models with Adaptive Momentum
• Authors: Ziqing Wen, Xiaoge Deng, Ping Luo, Tao Sun, Dongsheng Li,
• Abstract summary: We propose an adaptive momentum sampling method to accelerate the transforming process. We show that our method can produce more faithful images/graphs in small sampling steps with 2 to 5 times speed up.
• Score: 40.84399531998246
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Score-based generative models have demonstrated significant practical success in data-generating tasks. The models establish a diffusion process that perturbs the ground truth data to Gaussian noise and then learn the reverse process to transform noise into data. However, existing denoising methods such as Langevin dynamic and numerical stochastic differential equation solvers enjoy randomness but generate data slowly with a large number of score function evaluations, and the ordinary differential equation solvers enjoy faster sampling speed but no randomness may influence the sample quality. To this end, motivated by the Stochastic Gradient Descent (SGD) optimization methods and the high connection between the model sampling process with the SGD, we propose adaptive momentum sampling to accelerate the transforming process without introducing additional hyperparameters. Theoretically, we proved our method promises convergence under given conditions. In addition, we empirically show that our sampler can produce more faithful images/graphs in small sampling steps with 2 to 5 times speed up and obtain competitive scores compared to the baselines on image and graph generation tasks.

Related papers

• Neural Flow Samplers with Shortcut Models [19.81513273510523]
  Flow-based samplers generate samples by learning a velocity field that satisfies the continuity equation. While importance sampling provides an approximation, it suffers from high variance.
  arXiv  Detail & Related papers  (2025-02-11T07:55:41Z)
• Distributional Diffusion Models with Scoring Rules [83.38210785728994]
  Diffusion models generate high-quality synthetic data. generating high-quality outputs requires many discretization steps. We propose to accomplish sample generation by learning the posterior em distribution of clean data samples.
  arXiv  Detail & Related papers  (2025-02-04T16:59:03Z)
• A Geometric Perspective on Diffusion Models [57.27857591493788]
  We inspect the ODE-based sampling of a popular variance-exploding SDE. We establish a theoretical relationship between the optimal ODE-based sampling and the classic mean-shift (mode-seeking) algorithm.
  arXiv  Detail & Related papers  (2023-05-31T15:33:16Z)
• Parallel Sampling of Diffusion Models [76.3124029406809]
  Diffusion models are powerful generative models but suffer from slow sampling. We present ParaDiGMS, a novel method to accelerate the sampling of pretrained diffusion models by denoising multiple steps in parallel.
  arXiv  Detail & Related papers  (2023-05-25T17:59:42Z)
• Score-based diffusion models for accelerated MRI [35.3148116010546]
  We introduce a way to sample data from a conditional distribution given the measurements, such that the model can be readily used for solving inverse problems in imaging. Our model requires magnitude images only for training, and yet is able to reconstruct complex-valued data, and even extends to parallel imaging.
  arXiv  Detail & Related papers  (2021-10-08T08:42:03Z)
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed [0.0]
  Iterative generative models, such as noise conditional score networks, produce high quality samples by gradually denoising an initial noise vector. We establish a novel connection between knowledge distillation and image generation with a technique that distills a multi-step denoising process into a single step. Our Denoising Student generates high quality samples comparable to GANs on the CIFAR-10 and CelebA datasets, without adversarial training.
  arXiv  Detail & Related papers  (2021-01-07T06:12:28Z)
• Score-Based Generative Modeling through Stochastic Differential Equations [114.39209003111723]
  We present a differential equation that transforms a complex data distribution to a known prior distribution by injecting noise. A corresponding reverse-time SDE transforms the prior distribution back into the data distribution by slowly removing the noise. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks. We demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.
  arXiv  Detail & Related papers  (2020-11-26T19:39:10Z)
• Denoising Diffusion Implicit Models [117.03720513930335]
  We present denoising diffusion implicit models (DDIMs) for iterative implicit probabilistic models with the same training procedure as DDPMs. DDIMs can produce high quality samples $10 times$ to $50 times$ faster in terms of wall-clock time compared to DDPMs.
  arXiv  Detail & Related papers  (2020-10-06T06:15:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.