Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance

• URL: http://arxiv.org/abs/2410.13136v1
• Date: Thu, 17 Oct 2024 01:48:05 GMT
• Title: Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance
• Authors: Jiwan Hur, Dong-Jae Lee, Gyojin Han, Jaehyun Choi, Yunho Jeon, Junmo Kim,
• Abstract summary: Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps. We propose a self-guidance sampling method, which leads to better generation quality.
• Score: 25.41734642338575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps compared to continuous diffusion models. However, MGMs still underperform in image synthesis compared to recent well-developed continuous diffusion models with similar size in terms of quality and diversity of generated samples. A key factor in the performance of continuous diffusion models stems from the guidance methods, which enhance the sample quality at the expense of diversity. In this paper, we extend these guidance methods to generalized guidance formulation for MGMs and propose a self-guidance sampling method, which leads to better generation quality. The proposed approach leverages an auxiliary task for semantic smoothing in vector-quantized token space, analogous to the Gaussian blur in continuous pixel space. Equipped with the parameter-efficient fine-tuning method and high-temperature sampling, MGMs with the proposed self-guidance achieve a superior quality-diversity trade-off, outperforming existing sampling methods in MGMs with more efficient training and sampling costs. Extensive experiments with the various sampling hyperparameters confirm the effectiveness of the proposed self-guidance.

Related papers

• Accelerated Diffusion Models via Speculative Sampling [89.43940130493233]
  Speculative sampling is a popular technique for accelerating inference in Large Language Models. We extend speculative sampling to diffusion models, which generate samples via continuous, vector-valued Markov chains. We propose various drafting strategies, including a simple and effective approach that does not require training a draft model.
  arXiv  Detail & Related papers  (2025-01-09T16:50:16Z)
• Preference Optimization with Multi-Sample Comparisons [53.02717574375549]
  We introduce a novel approach that extends post-training to include multi-sample comparisons. These approaches fail to capture critical characteristics such as generative diversity and bias. We demonstrate that multi-sample comparison is more effective in optimizing collective characteristics than single-sample comparison.
  arXiv  Detail & Related papers  (2024-10-16T00:59:19Z)
• EM Distillation for One-step Diffusion Models [65.57766773137068]
  We propose a maximum likelihood-based approach that distills a diffusion model to a one-step generator model with minimal loss of quality. We develop a reparametrized sampling scheme and a noise cancellation technique that together stabilizes the distillation process.
  arXiv  Detail & Related papers  (2024-05-27T05:55:22Z)
• Diffusive Gibbs Sampling [40.1197715949575]
  We propose Diffusive Gibbs Sampling (DiGS) for effective sampling from distributions characterized by distant and disconnected modes. DiGS integrates recent developments in diffusion models, leveraging Gaussian convolution to create an auxiliary noisy distribution. A novel Metropolis-within-Gibbs scheme is proposed to enhance mixing in the denoising sampling step.
  arXiv  Detail & Related papers  (2024-02-05T13:47:41Z)
• Fast Sampling via Discrete Non-Markov Diffusion Models [49.598085130313514]
  We propose a discrete non-Markov diffusion model, which admits an accelerated reverse sampling for discrete data generation. Our method significantly reduces the number of function evaluations (i.e., calls to the neural network), making the sampling process much faster.
  arXiv  Detail & Related papers  (2023-12-14T18:14:11Z)
• Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [64.95663299945171]
  Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming. There exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. We propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs.
  arXiv  Detail & Related papers  (2023-09-10T22:05:24Z)
• Semi-Implicit Denoising Diffusion Models (SIDDMs) [50.30163684539586]
  Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. We introduce a novel approach that tackles the problem by matching implicit and explicit factors. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.
  arXiv  Detail & Related papers  (2023-06-21T18:49:22Z)
• Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
  We present MMD-DDM, a novel method for fast sampling of diffusion models. Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps. Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
  arXiv  Detail & Related papers  (2023-01-19T09:48:07Z)
• Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning [52.72369034247396]
  We propose the diffusion glancing transformer, which employs a modality diffusion process and residual glancing sampling. DIFFGLAT achieves better generation accuracy while maintaining fast decoding speed compared with both autoregressive and non-autoregressive models.
  arXiv  Detail & Related papers  (2022-12-20T13:36:25Z)
• Improving Sample Quality of Diffusion Models Using Self-Attention Guidance [36.42984435784378]
  Self-Attention Guidance (SAG) improves the performance of various diffusion models. SAG adversarially blurs only the regions that diffusion models attend to at each and guides them accordingly. Our results show that our SAG improves the performance of various diffusion models, including ADM, IDDPM, Stable Diffusion, and DiT.
  arXiv  Detail & Related papers  (2022-10-03T13:50:58Z)
• Refining Deep Generative Models via Discriminator Gradient Flow [18.406499703293566]
  Discriminator Gradient flow (DGflow) is a new technique that improves generated samples via the gradient flow of entropy-regularized f-divergences. We show that DGflow leads to significant improvement in the quality of generated samples for a variety of generative models.
  arXiv  Detail & Related papers  (2020-12-01T19:10:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.