SDEdit: Image Synthesis and Editing with Stochastic Differential Equations

• URL: http://arxiv.org/abs/2108.01073v1
• Date: Mon, 2 Aug 2021 17:59:47 GMT
• Title: SDEdit: Image Synthesis and Editing with Stochastic Differential Equations
• Authors: Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon
• Abstract summary: We introduce Differential Editing (SDEdit), based on a recent generative model using differential equations (SDEs) Given an input image with user edits, we first add noise to the input according to an SDE, and subsequently denoise it by simulating the reverse SDE to gradually increase its likelihood under the prior. Our method does not require task-specific loss function designs, which are critical components for recent image editing methods based on GAN inversions.
• Score: 113.35735935347465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a new image editing and synthesis framework, Stochastic Differential Editing (SDEdit), based on a recent generative model using stochastic differential equations (SDEs). Given an input image with user edits (e.g., hand-drawn color strokes), we first add noise to the input according to an SDE, and subsequently denoise it by simulating the reverse SDE to gradually increase its likelihood under the prior. Our method does not require task-specific loss function designs, which are critical components for recent image editing methods based on GAN inversion. Compared to conditional GANs, we do not need to collect new datasets of original and edited images for new applications. Therefore, our method can quickly adapt to various editing tasks at test time without re-training models. Our approach achieves strong performance on a wide range of applications, including image synthesis and editing guided by stroke paintings and image compositing.

Related papers

• Stable Flow: Vital Layers for Training-Free Image Editing [74.52248787189302]
  Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT) We propose an automatic method to identify "vital layers" within DiT, crucial for image formation. Next, to enable real-image editing, we introduce an improved image inversion method for flow models.
  arXiv  Detail & Related papers  (2024-11-21T18:59:51Z)
• CODE: Confident Ordinary Differential Editing [62.83365660727034]
  Confident Ordinary Differential Editing (CODE) is a novel approach for image synthesis that effectively handles Out-of-Distribution (OoD) guidance images. CODE enhances images through score-based updates along the probability-flow Ordinary Differential Equation (ODE) trajectory. Our method operates in a fully blind manner, relying solely on a pre-trained generative model.
  arXiv  Detail & Related papers  (2024-08-22T14:12:20Z)
• Diffusion Model-Based Image Editing: A Survey [46.244266782108234]
  Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks. We provide an exhaustive overview of existing methods using diffusion models for image editing. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval.
  arXiv  Detail & Related papers  (2024-02-27T14:07:09Z)
• Collaborative Score Distillation for Consistent Visual Synthesis [70.29294250371312]
  Collaborative Score Distillation (CSD) is based on the Stein Variational Gradient Descent (SVGD) We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-07-04T17:31:50Z)
• Image Restoration with Mean-Reverting Stochastic Differential Equations [9.245782611878752]
  This paper presents a differential equation (SDE) approach for general-purpose image restoration. By simulating the corresponding reverse-time SDE, we are able to restore the origin of the low-quality image. Experiments show that our proposed method achieves highly competitive performance in quantitative comparisons on image deraining, deblurring, and denoising.
  arXiv  Detail & Related papers  (2023-01-27T13:20:48Z)
• Delta-GAN-Encoder: Encoding Semantic Changes for Explicit Image Editing, using Few Synthetic Samples [2.348633570886661]
  We propose a novel method for learning to control any desired attribute in a pre-trained GAN's latent space. We perform Sim2Real learning, relying on minimal samples to achieve an unlimited amount of continuous precise edits.
  arXiv  Detail & Related papers  (2021-11-16T12:42:04Z)
• Score-Based Generative Modeling through Stochastic Differential Equations [114.39209003111723]
  We present a differential equation that transforms a complex data distribution to a known prior distribution by injecting noise. A corresponding reverse-time SDE transforms the prior distribution back into the data distribution by slowly removing the noise. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks. We demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.
  arXiv  Detail & Related papers  (2020-11-26T19:39:10Z)
• Semantic Photo Manipulation with a Generative Image Prior [86.01714863596347]
  GANs are able to synthesize images conditioned on inputs such as user sketch, text, or semantic labels. It is hard for GANs to precisely reproduce an input image. In this paper, we address these issues by adapting the image prior learned by GANs to image statistics of an individual image. Our method can accurately reconstruct the input image and synthesize new content, consistent with the appearance of the input image.
  arXiv  Detail & Related papers  (2020-05-15T18:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.