High-Fidelity Diffusion-based Image Editing
- URL: http://arxiv.org/abs/2312.15707v3
- Date: Thu, 4 Jan 2024 07:42:19 GMT
- Title: High-Fidelity Diffusion-based Image Editing
- Authors: Chen Hou, Guoqiang Wei, Zhibo Chen
- Abstract summary: The editing performance of diffusion models tends to be no more satisfactory even with increasing denoising steps.
We propose an innovative framework where a Markov module is incorporated to modulate diffusion model weights with residual features.
We introduce a novel learning paradigm aimed at minimizing error propagation during the editing process, which trains the editing procedure in a manner similar to denoising score-matching.
- Score: 19.85446433564999
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Diffusion models have attained remarkable success in the domains of image
generation and editing. It is widely recognized that employing larger inversion
and denoising steps in diffusion model leads to improved image reconstruction
quality. However, the editing performance of diffusion models tends to be no
more satisfactory even with increasing denoising steps. The deficiency in
editing could be attributed to the conditional Markovian property of the
editing process, where errors accumulate throughout denoising steps. To tackle
this challenge, we first propose an innovative framework where a rectifier
module is incorporated to modulate diffusion model weights with residual
features, thereby providing compensatory information to bridge the fidelity
gap. Furthermore, we introduce a novel learning paradigm aimed at minimizing
error propagation during the editing process, which trains the editing
procedure in a manner similar to denoising score-matching. Extensive
experiments demonstrate that our proposed framework and training strategy
achieve high-fidelity reconstruction and editing results across various levels
of denoising steps, meanwhile exhibits exceptional performance in terms of both
quantitative metric and qualitative assessments. Moreover, we explore our
model's generalization through several applications like image-to-image
translation and out-of-domain image editing.
Related papers
- FreeDiff: Progressive Frequency Truncation for Image Editing with Diffusion Models [44.26371926512843]
We introduce a novel free approach that employs progressive $textbfFre$qu$textbfe$ncy truncation to refine the guidance of $textbfDiff$usion models for universal editing tasks.
Our method achieves comparable results with state-of-the-art methods across a variety of editing tasks and on a diverse set of images.
arXiv Detail & Related papers (2024-04-18T04:47:28Z) - ReNoise: Real Image Inversion Through Iterative Noising [62.96073631599749]
We introduce an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations.
We evaluate the performance of our ReNoise technique using various sampling algorithms and models, including recent accelerated diffusion models.
arXiv Detail & Related papers (2024-03-21T17:52:08Z) - Diffusion Model-Based Image Editing: A Survey [46.244266782108234]
Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks.
We provide an exhaustive overview of existing methods using diffusion models for image editing.
To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval.
arXiv Detail & Related papers (2024-02-27T14:07:09Z) - Tuning-Free Inversion-Enhanced Control for Consistent Image Editing [44.311286151669464]
We present a novel approach called Tuning-free Inversion-enhanced Control (TIC)
TIC correlates features from the inversion process with those from the sampling process to mitigate the inconsistency in DDIM reconstruction.
We also propose a mask-guided attention concatenation strategy that combines contents from both the inversion and the naive DDIM editing processes.
arXiv Detail & Related papers (2023-12-22T11:13:22Z) - Reconstruct-and-Generate Diffusion Model for Detail-Preserving Image
Denoising [16.43285056788183]
We propose a novel approach called the Reconstruct-and-Generate Diffusion Model (RnG)
Our method leverages a reconstructive denoising network to recover the majority of the underlying clean signal.
It employs a diffusion algorithm to generate residual high-frequency details, thereby enhancing visual quality.
arXiv Detail & Related papers (2023-09-19T16:01:20Z) - Gradpaint: Gradient-Guided Inpainting with Diffusion Models [71.47496445507862]
Denoising Diffusion Probabilistic Models (DDPMs) have recently achieved remarkable results in conditional and unconditional image generation.
We present GradPaint, which steers the generation towards a globally coherent image.
We generalizes well to diffusion models trained on various datasets, improving upon current state-of-the-art supervised and unsupervised methods.
arXiv Detail & Related papers (2023-09-18T09:36:24Z) - Steerable Conditional Diffusion for Out-of-Distribution Adaptation in
Imaging Inverse Problems [78.76955228709241]
We introduce a novel sampling framework called Steerable Conditional Diffusion.
This framework adapts the denoising network specifically to the available measured data.
We achieve substantial enhancements in OOD performance across diverse imaging modalities.
arXiv Detail & Related papers (2023-08-28T08:47:06Z) - Stimulating the Diffusion Model for Image Denoising via Adaptive Embedding and Ensembling [56.506240377714754]
We present a novel strategy called the Diffusion Model for Image Denoising (DMID)
Our strategy includes an adaptive embedding method that embeds the noisy image into a pre-trained unconditional diffusion model.
Our DMID strategy achieves state-of-the-art performance on both distortion-based and perception-based metrics.
arXiv Detail & Related papers (2023-07-08T14:59:41Z) - DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly
Detection [89.49600182243306]
We reformulate the reconstruction process using a diffusion model into a noise-to-norm paradigm.
We propose a rapid one-step denoising paradigm, significantly faster than the traditional iterative denoising in diffusion models.
The segmentation sub-network predicts pixel-level anomaly scores using the input image and its anomaly-free restoration.
arXiv Detail & Related papers (2023-03-15T16:14:06Z) - End-to-End Visual Editing with a Generatively Pre-Trained Artist [78.5922562526874]
We consider the targeted image editing problem: blending a region in a source image with a driver image that specifies the desired change.
We propose a self-supervised approach that simulates edits by augmenting off-the-shelf images in a target domain.
We show that different blending effects can be learned by an intuitive control of the augmentation process, with no other changes required to the model architecture.
arXiv Detail & Related papers (2022-05-03T17:59:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.