Not All Steps are Created Equal: Selective Diffusion Distillation for Image Manipulation

• URL: http://arxiv.org/abs/2307.08448v1
• Date: Mon, 17 Jul 2023 12:42:56 GMT
• Title: Not All Steps are Created Equal: Selective Diffusion Distillation for Image Manipulation
• Authors: Luozhou Wang, Shuai Yang, Shu Liu, Ying-cong Chen
• Abstract summary: Conditional diffusion models have demonstrated impressive performance in image manipulation tasks. Adding too much noise affects the fidelity of the image while adding too little affects its editability. We propose a novel framework, Diffusion Selective Distillation (SDD), that ensures both the fidelity and editability of images.
• Score: 23.39614544877529
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conditional diffusion models have demonstrated impressive performance in image manipulation tasks. The general pipeline involves adding noise to the image and then denoising it. However, this method faces a trade-off problem: adding too much noise affects the fidelity of the image while adding too little affects its editability. This largely limits their practical applicability. In this paper, we propose a novel framework, Selective Diffusion Distillation (SDD), that ensures both the fidelity and editability of images. Instead of directly editing images with a diffusion model, we train a feedforward image manipulation network under the guidance of the diffusion model. Besides, we propose an effective indicator to select the semantic-related timestep to obtain the correct semantic guidance from the diffusion model. This approach successfully avoids the dilemma caused by the diffusion process. Our extensive experiments demonstrate the advantages of our framework. Code is released at https://github.com/AndysonYs/Selective-Diffusion-Distillation.

Related papers

• Reinforced Diffusion: Learning to Push the Limits of Anisotropic Diffusion for Image Denoising [57.226775716102765]
  We describe a trainable anisotropic diffusion framework based on reinforcement learning.<n>By modeling the denoising process as a series of naive diffusion actions with iterations order learned by deep Q-learning, we propose an effective diffusion-based image denoiser.
  arXiv  Detail & Related papers  (2025-12-30T07:23:15Z)
• Towards Reliable Identification of Diffusion-based Image Manipulations [29.011252426887577]
  We propose a novel approach for ReliAble iDentification of inpainted AReas (RADAR)<n>RADAR builds on existing foundation models and combines features from different image modalities.<n>Our experiments show that RADAR achieves excellent results, outperforming the state-of-the-art in detecting and localising image edits.
  arXiv  Detail & Related papers  (2025-06-05T18:00:01Z)
• Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment [56.609042046176555]
  suboptimal noise-data mapping leads to slow training of diffusion models. Drawing inspiration from the immiscibility phenomenon in physics, we propose Immiscible Diffusion. Our approach is remarkably simple, requiring only one line of code to restrict the diffuse-able area for each image.
  arXiv  Detail & Related papers  (2024-06-18T06:20:42Z)
• Diffusion Models With Learned Adaptive Noise [12.530583016267768]
  In this paper, we explore whether the diffusion process can be learned from data. A widely held assumption is that the ELBO is invariant to the noise process. We propose MULAN, a learned diffusion process that applies noise at different rates across an image.
  arXiv  Detail & Related papers  (2023-12-20T18:00:16Z)
• Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models [82.8261101680427]
  Smooth latent spaces ensure that a perturbation on an input latent corresponds to a steady change in the output image. This property proves beneficial in downstream tasks, including image inversion, inversion, and editing. We propose Smooth Diffusion, a new category of diffusion models that can be simultaneously high-performing and smooth.
  arXiv  Detail & Related papers  (2023-12-07T16:26:23Z)
• Fuzzy-Conditioned Diffusion and Diffusion Projection Attention Applied to Facial Image Correction [14.34815548338413]
  We derive a fuzzy-conditioned diffusion, where implicit diffusion priors can be exploited with controllable strength. We propose an application to facial image correction, where we combine our fuzzy-conditioned diffusion with diffusion-derived attention maps.
  arXiv  Detail & Related papers  (2023-06-26T17:58:00Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• Diffusion Visual Counterfactual Explanations [51.077318228247925]
  Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts. In this paper, we overcome this by generating Visual Diffusion Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers.
  arXiv  Detail & Related papers  (2022-10-21T09:35:47Z)
• Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise [52.59444045853966]
  We show that an entire family of generative models can be constructed by varying the choice of image degradation. The success of fully deterministic models calls into question the community's understanding of diffusion models.
  arXiv  Detail & Related papers  (2022-08-19T15:18:39Z)
• Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image Synthesis [39.671396431940224]
  diffusion models have shown remarkable results in image synthesis by gradually removing noise and amplifying signals. We propose a novel generative process that synthesizes images in a coarse-to-fine manner. Experiments show that the proposed model outperforms the previous method in FID on LSUN bedroom and church datasets.
  arXiv  Detail & Related papers  (2022-07-16T15:00:21Z)
• Non Gaussian Denoising Diffusion Models [91.22679787578438]
  We show that noise from Gamma distribution provides improved results for image and speech generation. We also show that using a mixture of Gaussian noise variables in the diffusion process improves the performance over a diffusion process that is based on a single distribution.
  arXiv  Detail & Related papers  (2021-06-14T16:42:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.