The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

• URL: http://arxiv.org/abs/2311.01410v2
• Date: Thu, 29 Feb 2024 14:31:13 GMT
• Title: The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing
• Authors: Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, Chongxuan Li
• Abstract summary: We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner. We provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation. We also propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging.
• Score: 24.76844856196734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed DragBench) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.

Related papers

• Mechanistic PDE Networks for Discovery of Governing Equations [52.492158106791365]
  We present Mechanistic PDE Networks, a model for discovery of partial differential equations from data. The represented PDEs are then solved and decoded for specific tasks. We develop a native, GPU-capable, parallel, sparse, and differentiable multigrid solver specialized for linear partial differential equations.
  arXiv  Detail & Related papers  (2025-02-25T17:21:44Z)
• CODE: Confident Ordinary Differential Editing [62.83365660727034]
  Confident Ordinary Differential Editing (CODE) is a novel approach for image synthesis that effectively handles Out-of-Distribution (OoD) guidance images. CODE enhances images through score-based updates along the probability-flow Ordinary Differential Equation (ODE) trajectory. Our method operates in a fully blind manner, relying solely on a pre-trained generative model.
  arXiv  Detail & Related papers  (2024-08-22T14:12:20Z)
• Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers [55.0876373185983]
  We present the Universal PDE solver (Unisolver) capable of solving a wide scope of PDEs. Our key finding is that a PDE solution is fundamentally under the control of a series of PDE components. Unisolver achieves consistent state-of-the-art results on three challenging large-scale benchmarks.
  arXiv  Detail & Related papers  (2024-05-27T15:34:35Z)
• AdjointDEIS: Efficient Gradients for Diffusion Models [2.0795007613453445]
  We show that continuous adjoint equations for diffusion SDEs actually simplify to a simple ODE. We also demonstrate the effectiveness of AdjointDEIS for guided generation with an adversarial attack in the form of the face morphing problem.
  arXiv  Detail & Related papers  (2024-05-23T19:51:33Z)
• Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing [58.48890547818074]
  We present a powerful modification of Contrastive Denoising Score (CUT) for latent diffusion models (LDM) Our approach enables zero-shot imageto-image translation and neural field (NeRF) editing, achieving structural correspondence between the input and output.
  arXiv  Detail & Related papers  (2023-11-30T15:06:10Z)
• Closing the ODE-SDE gap in score-based diffusion models through the Fokker-Planck equation [0.562479170374811]
  We rigorously describe the range of dynamics and approximations that arise when training score-based diffusion models. We show numerically that conventional score-based diffusion models can exhibit significant differences between ODE- and SDE-induced distributions.
  arXiv  Detail & Related papers  (2023-11-27T16:44:50Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• Elucidating the solution space of extended reverse-time SDE for diffusion models [54.23536653351234]
  Diffusion models (DMs) demonstrate potent image generation capabilities in various generative modeling tasks. Their primary limitation lies in slow sampling speed, requiring hundreds or thousands of sequential function evaluations to generate high-quality images. We formulate the sampling process as an extended reverse-time SDE, unifying prior explorations into ODEs and SDEs. We devise fast and training-free samplers, ER-SDE-rs, achieving state-of-the-art performance across all samplers.
  arXiv  Detail & Related papers  (2023-09-12T12:27:17Z)
• Image Restoration with Mean-Reverting Stochastic Differential Equations [9.245782611878752]
  This paper presents a differential equation (SDE) approach for general-purpose image restoration. By simulating the corresponding reverse-time SDE, we are able to restore the origin of the low-quality image. Experiments show that our proposed method achieves highly competitive performance in quantitative comparisons on image deraining, deblurring, and denoising.
  arXiv  Detail & Related papers  (2023-01-27T13:20:48Z)
• SDEdit: Image Synthesis and Editing with Stochastic Differential Equations [113.35735935347465]
  We introduce Differential Editing (SDEdit), based on a recent generative model using differential equations (SDEs) Given an input image with user edits, we first add noise to the input according to an SDE, and subsequently denoise it by simulating the reverse SDE to gradually increase its likelihood under the prior. Our method does not require task-specific loss function designs, which are critical components for recent image editing methods based on GAN inversions.
  arXiv  Detail & Related papers  (2021-08-02T17:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.