Diffusion Models as Cartoonists: The Curious Case of High Density Regions

• URL: http://arxiv.org/abs/2411.01293v3
• Date: Wed, 12 Mar 2025 12:08:55 GMT
• Title: Diffusion Models as Cartoonists: The Curious Case of High Density Regions
• Authors: Rafał Karczewski, Markus Heinonen, Vikas Garg,
• Abstract summary: We investigate what kind of images lie in the high-density regions of diffusion models.<n>We propose a practical high-density sampler that consistently generates images of higher likelihood than usual samplers.
• Score: 12.875154616215305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate what kind of images lie in the high-density regions of diffusion models. We introduce a theoretical mode-tracking process capable of pinpointing the exact mode of the denoising distribution, and we propose a practical high-density sampler that consistently generates images of higher likelihood than usual samplers. Our empirical findings reveal the existence of significantly higher likelihood samples that typical samplers do not produce, often manifesting as cartoon-like drawings or blurry images depending on the noise level. Curiously, these patterns emerge in datasets devoid of such examples. We also present a novel approach to track sample likelihoods in diffusion SDEs, which remarkably incurs no additional computational cost.

Related papers

• Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models [76.46246743508651]
  We show that current diffusion models actually have an expressive bottleneck in backward denoising. We introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising.
  arXiv  Detail & Related papers  (2023-09-25T12:03:32Z)
• Semi-Implicit Denoising Diffusion Models (SIDDMs) [50.30163684539586]
  Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. We introduce a novel approach that tackles the problem by matching implicit and explicit factors. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.
  arXiv  Detail & Related papers  (2023-06-21T18:49:22Z)
• Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision [76.32860119056964]
  We propose a novel class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never directly observed. We demonstrate the effectiveness of our method on three challenging computer vision tasks.
  arXiv  Detail & Related papers  (2023-06-20T17:53:00Z)
• Denoising Diffusion Samplers [41.796349001299156]
  Denoising diffusion models are a popular class of generative models providing state-of-the-art results in many domains. We explore a similar idea to sample approximately from unnormalized probability density functions and estimate their normalizing constants. While score matching is not applicable in this context, we can leverage many of the ideas introduced in generative modeling for Monte Carlo sampling.
  arXiv  Detail & Related papers  (2023-02-27T14:37:16Z)
• Example-Based Sampling with Diffusion Models [7.943023838493658]
  diffusion models for image generation could be appropriate for learning how to generate point sets from examples. We propose a generic way to produce 2-d point sets imitating existing samplers from observed point sets using a diffusion model. We demonstrate how the differentiability of our approach can be used to optimize point sets to enforce properties.
  arXiv  Detail & Related papers  (2023-02-10T08:35:17Z)
• ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectories [144.03939123870416]
  We propose a novel conditional diffusion model by introducing conditions into the forward process. We use extra latent space to allocate an exclusive diffusion trajectory for each condition based on some shifting rules. We formulate our method, which we call textbfShiftDDPMs, and provide a unified point of view on existing related methods.
  arXiv  Detail & Related papers  (2023-02-05T12:48:21Z)
• Don't Play Favorites: Minority Guidance for Diffusion Models [59.75996752040651]
  We present a novel framework that can make the generation process of the diffusion models focus on the minority samples. We develop minority guidance, a sampling technique that can guide the generation process toward regions with desired likelihood levels.
  arXiv  Detail & Related papers  (2023-01-29T03:08:47Z)
• Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
  We present MMD-DDM, a novel method for fast sampling of diffusion models. Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps. Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
  arXiv  Detail & Related papers  (2023-01-19T09:48:07Z)
• Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors [64.24948495708337]
  We introduce a new method that brings predicted samples to the training data manifold using a pretrained unconditional diffusion model. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks.
  arXiv  Detail & Related papers  (2022-12-14T17:26:35Z)
• Generating High Fidelity Data from Low-density Regions using Diffusion Models [15.819414178363571]
  We leverage diffusion process based generative models to synthesize novel images from low-density regions. We modify the sampling process to guide it towards low-density regions while simultaneously maintaining the fidelity of synthetic data.
  arXiv  Detail & Related papers  (2022-03-31T17:56:25Z)
• Learning Energy-Based Models by Diffusion Recovery Likelihood [61.069760183331745]
  We present a diffusion recovery likelihood method to tractably learn and sample from a sequence of energy-based models. After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs.
  arXiv  Detail & Related papers  (2020-12-15T07:09:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.