OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models

• URL: http://arxiv.org/abs/2306.08860v1
• Date: Thu, 15 Jun 2023 05:04:39 GMT
• Title: OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models
• Authors: Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang and Yu Wang
• Abstract summary: Diffusion probabilistic models (DPMs) are a new class of generative models that have achieved state-of-the-art generation quality. One major drawback of DPMs is the slow generation speed due to the large number of neural network evaluations required in the generation process. We design OMS-DPM, a predictor-based search algorithm, to optimize the model schedule given an arbitrary generation time budget and a set of pre-trained models.
• Score: 14.247927015966791
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion probabilistic models (DPMs) are a new class of generative models that have achieved state-of-the-art generation quality in various domains. Despite the promise, one major drawback of DPMs is the slow generation speed due to the large number of neural network evaluations required in the generation process. In this paper, we reveal an overlooked dimension -- model schedule -- for optimizing the trade-off between generation quality and speed. More specifically, we observe that small models, though having worse generation quality when used alone, could outperform large models in certain generation steps. Therefore, unlike the traditional way of using a single model, using different models in different generation steps in a carefully designed \emph{model schedule} could potentially improve generation quality and speed \emph{simultaneously}. We design OMS-DPM, a predictor-based search algorithm, to optimize the model schedule given an arbitrary generation time budget and a set of pre-trained models. We demonstrate that OMS-DPM can find model schedules that improve generation quality and speed than prior state-of-the-art methods across CIFAR-10, CelebA, ImageNet, and LSUN datasets. When applied to the public checkpoints of the Stable Diffusion model, we are able to accelerate the sampling by 2$\times$ while maintaining the generation quality.

Related papers

• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step. Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• Flow Generator Matching [35.371071097381346]
  Flow Generator Matching (FGM) is designed to accelerate the sampling of flow-matching models into a one-step generation. On the CIFAR10 unconditional generation benchmark, our one-step FGM model achieves a new record Fr'echet Inception Distance (FID) score of 3.08. MM-DiT-FGM one-step text-to-image model demonstrates outstanding industry-level performance.
  arXiv  Detail & Related papers  (2024-10-25T05:41:28Z)
• Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models [69.06149482021071]
  We propose a novel EHR data generation model called EHRPD. It is a diffusion-based model designed to predict the next visit based on the current one while also incorporating time interval estimation. We conduct experiments on two public datasets and evaluate EHRPD from fidelity, privacy, and utility perspectives.
  arXiv  Detail & Related papers  (2024-06-20T02:20:23Z)
• MGE: A Training-Free and Efficient Model Generation and Enhancement Scheme [10.48591131837771]
  This paper proposes a Training-Free and Efficient Model Generation and Enhancement Scheme (MGE) It considers two aspects during the model generation process: the distribution of model parameters and model performance. Experiments result shows that generated models are comparable to models obtained through normal training, and even superior in some cases.
  arXiv  Detail & Related papers  (2024-02-27T13:12:00Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Generative Marginalization Models [21.971818180264943]
  marginalization models (MAMs) are a new family of generative models for high-dimensional discrete data. They offer scalable and flexible generative modeling by explicitly modeling all induced marginal distributions. For energy-based training tasks, MAMs enable any-order generative modeling of high-dimensional problems beyond the scale of previous methods.
  arXiv  Detail & Related papers  (2023-10-19T17:14:29Z)
• Stable Training of Probabilistic Models Using the Leave-One-Out Maximum Log-Likelihood Objective [0.7373617024876725]
  Kernel density estimation (KDE) based models are popular choices for this task, but they fail to adapt to data regions with varying densities. An adaptive KDE model is employed to circumvent this, where each kernel in the model has an individual bandwidth. A modified expectation-maximization algorithm is employed to accelerate the optimization speed reliably.
  arXiv  Detail & Related papers  (2023-10-05T14:08:42Z)
• Consistency Models [89.68380014789861]
  We propose a new family of models that generate high quality samples by directly mapping noise to data. They support fast one-step generation by design, while still allowing multistep sampling to trade compute for sample quality. They also support zero-shot data editing, such as image inpainting, colorization, and super-resolution, without requiring explicit training.
  arXiv  Detail & Related papers  (2023-03-02T18:30:16Z)
• Learning to Efficiently Sample from Diffusion Probabilistic Models [49.58748345998702]
  Denoising Diffusion Probabilistic Models (DDPMs) can yield high-fidelity samples and competitive log-likelihoods across a range of domains. We introduce an exact dynamic programming algorithm that finds the optimal discrete time schedules for any pre-trained DDPM.
  arXiv  Detail & Related papers  (2021-06-07T17:15:07Z)
• Dynamic Model Pruning with Feedback [64.019079257231]
  We propose a novel model compression method that generates a sparse trained model without additional overhead. We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models.
  arXiv  Detail & Related papers  (2020-06-12T15:07:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.