Manifold Preserving Guided Diffusion

• URL: http://arxiv.org/abs/2311.16424v1
• Date: Tue, 28 Nov 2023 02:08:06 GMT
• Title: Manifold Preserving Guided Diffusion
• Authors: Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, Stefano Ermon
• Abstract summary: Conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. We propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework.
• Score: 121.97907811212123
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.

Related papers

• Constrained Diffusion with Trust Sampling [11.354281911272864]
  We rethink training-free loss-guided diffusion from an optimization perspective. Trust sampling effectively balances following the unconditional diffusion model and adhering to the loss guidance. We demonstrate the efficacy of our method through extensive experiments on complex tasks, and in drastically different domains of images and 3D motion generation.
  arXiv  Detail & Related papers  (2024-11-17T01:34:57Z)
• Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization [55.14484317645865]
  We develop a conditional diffusion model to produce exceptional quality prompts for offline reinforcement learning tasks. We show that the Prompt diffuser is a robust and effective tool for the prompt-tuning process, demonstrating strong performance in the meta-RL tasks.
  arXiv  Detail & Related papers  (2024-11-02T07:38:02Z)
• Dreamguider: Improved Training free Diffusion-based Conditional Generation [31.68823843900196]
  Dreamguider is a method that enables inference-time guidance without compute-heavy backpropagation through the diffusion network. We present experiments using Dreamguider on multiple tasks across multiple datasets and models to show the effectiveness of the proposed modules.
  arXiv  Detail & Related papers  (2024-06-04T17:59:32Z)
• Improved off-policy training of diffusion samplers [93.66433483772055]
  We study the problem of training diffusion models to sample from a distribution with an unnormalized density or energy function. We benchmark several diffusion-structured inference methods, including simulation-based variational approaches and off-policy methods. Our results shed light on the relative advantages of existing algorithms while bringing into question some claims from past work.
  arXiv  Detail & Related papers  (2024-02-07T18:51:49Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance [1.6804613362826175]
  Diffusion models have emerged as a pivotal advancement in generative models. In this paper we aim to underscore a discrepancy between conventional training methods and the desired conditional sampling behavior. We introduce an updated loss function that better aligns training objectives with sampling behaviors.
  arXiv  Detail & Related papers  (2023-11-02T02:03:12Z)
• Elucidating The Design Space of Classifier-Guided Diffusion Generation [17.704873767509557]
  We show that it is possible to achieve significant performance improvements over existing guidance schemes by leveraging off-the-shelf classifiers in a training-free fashion. Our proposed approach has great potential and can be readily scaled up to text-to-image generation tasks.
  arXiv  Detail & Related papers  (2023-10-17T14:34:58Z)
• Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models [63.1637853118899]
  We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models. We employ a shift control module that works on h-space of pre-trained diffusion models to manipulate a sample into a shifted version of itself. By jointly optimizing them, the model will spontaneously discover disentangled and interpretable directions.
  arXiv  Detail & Related papers  (2023-10-15T18:44:30Z)
• Structural Pruning for Diffusion Models [65.02607075556742]
  We present Diff-Pruning, an efficient compression method tailored for learning lightweight diffusion models from pre-existing ones. Our empirical assessment, undertaken across several datasets highlights two primary benefits of our proposed method.
  arXiv  Detail & Related papers  (2023-05-18T12:38:21Z)
• Towards Controllable Diffusion Models via Reward-Guided Exploration [15.857464051475294]
  We propose a novel framework that guides the training-phase of diffusion models via reinforcement learning (RL) RL enables calculating policy gradients via samples from a pay-off distribution proportional to exponential scaled rewards, rather than from policies themselves. Experiments on 3D shape and molecule generation tasks show significant improvements over existing conditional diffusion models.
  arXiv  Detail & Related papers  (2023-04-14T13:51:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.