Repulsive Score Distillation for Diverse Sampling of Diffusion Models

• URL: http://arxiv.org/abs/2406.16683v1
• Date: Mon, 24 Jun 2024 14:43:02 GMT
• Title: Repulsive Score Distillation for Diverse Sampling of Diffusion Models
• Authors: Nicolas Zilberstein, Morteza Mardani, Santiago Segarra,
• Abstract summary: Repulsive Score Distillation (RSD) is a variational framework based on repulsion of an ensemble of particles that promotes diversity. RSD achieves a superior trade-off between diversity and quality compared with state-of-the-art alternatives.
• Score: 31.255943277671893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Score distillation sampling has been pivotal for integrating diffusion models into generation of complex visuals. Despite impressive results it suffers from mode collapse and lack of diversity. To cope with this challenge, we leverage the gradient flow interpretation of score distillation to propose Repulsive Score Distillation (RSD). In particular, we propose a variational framework based on repulsion of an ensemble of particles that promotes diversity. Using a variational approximation that incorporates a coupling among particles, the repulsion appears as a simple regularization that allows interaction of particles based on their relative pairwise similarity, measured e.g., via radial basis kernels. We design RSD for both unconstrained and constrained sampling scenarios. For constrained sampling we focus on inverse problems in the latent space that leads to an augmented variational formulation, that strikes a good balance between compute, quality and diversity. Our extensive experiments for text-to-image generation, and inverse problems demonstrate that RSD achieves a superior trade-off between diversity and quality compared with state-of-the-art alternatives.

Related papers

• Diffusing Differentiable Representations [60.72992910766525]
  We introduce a novel, training-free method for sampling differentiable representations (diffreps) using pretrained diffusion models. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects.
  arXiv  Detail & Related papers  (2024-12-09T20:42:58Z)
• Diffusion-PINN Sampler [6.656265182236135]
  We introduce a novel diffusion-based sampling algorithm that estimates the drift term by solving the governing partial differential equation of the log-density of the underlying SDE marginals via physics-informed neural networks (PINN) We prove that the error of log-density approximation can be controlled by the PINN residual loss, enabling us to establish convergence guarantees of DPS.
  arXiv  Detail & Related papers  (2024-10-20T09:02:16Z)
• Total Uncertainty Quantification in Inverse PDE Solutions Obtained with Reduced-Order Deep Learning Surrogate Models [50.90868087591973]
  We propose an approximate Bayesian method for quantifying the total uncertainty in inverse PDE solutions obtained with machine learning surrogate models. We test the proposed framework by comparing it with the iterative ensemble smoother and deep ensembling methods for a non-linear diffusion equation.
  arXiv  Detail & Related papers  (2024-08-20T19:06:02Z)
• Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems [12.482127049881026]
  We propose a novel approach to solve inverse problems with a diffusion prior from an amortized variational inference perspective. Our amortized inference learns a function that directly maps measurements to the implicit posterior distributions of corresponding clean data, enabling a single-step posterior sampling even for unseen measurements.
  arXiv  Detail & Related papers  (2024-07-23T02:14:18Z)
• Stability and Generalizability in SDE Diffusion Models with Measure-Preserving Dynamics [11.919291977879801]
  Inverse problems describe the process of estimating the causal factors from a set of measurements or data. Diffusion models have shown promise as potent generative tools for solving inverse problems.
  arXiv  Detail & Related papers  (2024-06-19T15:55:12Z)
• Deep Data Consistency: a Fast and Robust Diffusion Model-based Solver for Inverse Problems [0.0]
  We propose Deep Data Consistency (DDC) to update the data consistency step with a deep learning model when solving inverse problems with diffusion models. In comparison with state-of-the-art methods in linear and non-linear tasks, DDC demonstrates its outstanding performance of both similarity and realness metrics.
  arXiv  Detail & Related papers  (2024-05-17T12:54:43Z)
• Distilling Diffusion Models into Conditional GANs [90.76040478677609]
  We distill a complex multistep diffusion model into a single-step conditional GAN student model. For efficient regression loss, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models.
  arXiv  Detail & Related papers  (2024-05-09T17:59:40Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• Semi-Implicit Denoising Diffusion Models (SIDDMs) [50.30163684539586]
  Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. We introduce a novel approach that tackles the problem by matching implicit and explicit factors. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.
  arXiv  Detail & Related papers  (2023-06-21T18:49:22Z)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models [101.831766524264]
  Inverse tasks can be formulated as inferring a posterior distribution over data. This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. We propose a variational approach that by design seeks to approximate the true posterior distribution.
  arXiv  Detail & Related papers  (2023-05-07T23:00:47Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.