Speed up the inference of diffusion models via shortcut MCMC sampling

• URL: http://arxiv.org/abs/2301.01206v1
• Date: Sun, 18 Dec 2022 07:37:26 GMT
• Title: Speed up the inference of diffusion models via shortcut MCMC sampling
• Authors: Gang Chen
• Abstract summary: Diffusion probabilistic models have generated high quality image synthesis recently. One pain point is the notorious inference to gradually obtain clear images with thousands of steps. We present a shortcut MCMC sampling algorithm, which balances training and inference, while keeping the generated data's quality.
• Score: 4.982806898121435
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Diffusion probabilistic models have generated high quality image synthesis recently. However, one pain point is the notorious inference to gradually obtain clear images with thousands of steps, which is time consuming compared to other generative models. In this paper, we present a shortcut MCMC sampling algorithm, which balances training and inference, while keeping the generated data's quality. In particular, we add the global fidelity constraint with shortcut MCMC sampling to combat the local fitting from diffusion models. We do some initial experiments and show very promising results. Our implementation is available at https://github.com//vividitytech/diffusion-mcmc.git.

Related papers

• Fast constrained sampling in pre-trained diffusion models [77.21486516041391]
  We propose an algorithm that enables fast and high-quality generation under arbitrary constraints. During inference, we can interchange between gradient updates computed on the noisy image and updates computed on the final, clean image. Our approach produces results that rival or surpass the state-of-the-art training-free inference approaches.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Truncated Consistency Models [57.50243901368328]
  Training consistency models requires learning to map all intermediate points along PF ODE trajectories to their corresponding endpoints. We empirically find that this training paradigm limits the one-step generation performance of consistency models. We propose a new parameterization of the consistency function and a two-stage training procedure that prevents the truncated-time training from collapsing to a trivial solution.
  arXiv  Detail & Related papers  (2024-10-18T22:38:08Z)
• Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning [45.89372687373466]
  Diffusion models are trained by learning a sequence of models that reverse each step of noise corruption. The parameters are fully shared across multiple timesteps to enhance training efficiency. However, since the denoising tasks differ at each timestep, the gradients computed at different timesteps may conflict, potentially degrading the overall performance of image generation.
  arXiv  Detail & Related papers  (2024-10-09T08:19:25Z)
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference [60.32804641276217]
  We propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs. A high-quality 768 x 768 24-step LCM takes only 32 A100 GPU hours for training. We also introduce Latent Consistency Fine-tuning (LCF), a novel method that is tailored for fine-tuning LCMs on customized image datasets.
  arXiv  Detail & Related papers  (2023-10-06T17:11:58Z)
• Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
  We present MMD-DDM, a novel method for fast sampling of diffusion models. Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps. Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
  arXiv  Detail & Related papers  (2023-01-19T09:48:07Z)
• Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models [54.1843419649895]
  We propose a solution based on denoising diffusion probabilistic models (DDPMs) Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task.
  arXiv  Detail & Related papers  (2022-12-01T18:59:55Z)
• Denoising MCMC for Accelerating Diffusion-Based Generative Models [54.06799491319278]
  Diffusion models are powerful generative models that simulate the reverse of diffusion processes using score functions to synthesize data from noise. Here, we propose an approach to accelerating score-based sampling: Denoising MCMC. We show that Denoising Langevin Gibbs (DLG), an instance of DMCMC, successfully accelerates all six reverse-S/ODE computation tasks.
  arXiv  Detail & Related papers  (2022-09-29T07:16:10Z)
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning [90.02873747873444]
  Bit Diffusion is a generic approach for generating discrete data with continuous diffusion models. The proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.
  arXiv  Detail & Related papers  (2022-08-08T15:08:40Z)
• Improving Diffusion Model Efficiency Through Patching [0.0]
  We find that adding a simple ViT-style patching transformation can considerably reduce a diffusion model's sampling time and memory usage. We justify our approach both through an analysis of diffusion model objective, and through empirical experiments on LSUN Church, ImageNet 256, and FFHQ 1024.
  arXiv  Detail & Related papers  (2022-07-09T18:21:32Z)
• Improved Denoising Diffusion Probabilistic Models [4.919647298882951]
  We show that DDPMs can achieve competitive log-likelihoods while maintaining high sample quality. We also find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes. We show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable.
  arXiv  Detail & Related papers  (2021-02-18T23:44:17Z)
• Denoising Diffusion Implicit Models [117.03720513930335]
  We present denoising diffusion implicit models (DDIMs) for iterative implicit probabilistic models with the same training procedure as DDPMs. DDIMs can produce high quality samples $10 times$ to $50 times$ faster in terms of wall-clock time compared to DDPMs.
  arXiv  Detail & Related papers  (2020-10-06T06:15:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.