Minutes to Seconds: Speeded-up DDPM-based Image Inpainting with Coarse-to-Fine Sampling

• URL: http://arxiv.org/abs/2407.05875v1
• Date: Mon, 8 Jul 2024 12:33:54 GMT
• Title: Minutes to Seconds: Speeded-up DDPM-based Image Inpainting with Coarse-to-Fine Sampling
• Authors: Lintao Zhang, Xiangcheng Du, LeoWu TomyEnrique, Yiqun Wang, Yingbin Zheng, Cheng Jin,
• Abstract summary: We propose an efficient DDPM-based image inpainting method which includes three speed-up strategies. First, we utilize a pre-trained Light-Weight Diffusion Model (LWDM) to reduce the number of parameters. Second, we introduce a skip-step sampling scheme of Denoising Diffusion Implicit Models (DDIM) for the denoising process.
• Score: 8.965432123669167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For image inpainting, the existing Denoising Diffusion Probabilistic Model (DDPM) based method i.e. RePaint can produce high-quality images for any inpainting form. It utilizes a pre-trained DDPM as a prior and generates inpainting results by conditioning on the reverse diffusion process, namely denoising process. However, this process is significantly time-consuming. In this paper, we propose an efficient DDPM-based image inpainting method which includes three speed-up strategies. First, we utilize a pre-trained Light-Weight Diffusion Model (LWDM) to reduce the number of parameters. Second, we introduce a skip-step sampling scheme of Denoising Diffusion Implicit Models (DDIM) for the denoising process. Finally, we propose Coarse-to-Fine Sampling (CFS), which speeds up inference by reducing image resolution in the coarse stage and decreasing denoising timesteps in the refinement stage. We conduct extensive experiments on both faces and general-purpose image inpainting tasks, and our method achieves competitive performance with approximately 60 times speedup.

Related papers

• Quick Bypass Mechanism of Zero-Shot Diffusion-Based Image Restoration [0.8192907805418583]
  We propose a strategy that accelerates the denoising process by initializing from an intermediate approximation, effectively bypassing early denoising steps.<n>We validate proposed methods on ImageNet-1K and CelebAHQ across multiple image restoration tasks, e.g., super-resolution, deblurring, and compressed sensing.
  arXiv  Detail & Related papers  (2025-07-06T01:36:27Z)
• Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy [44.09909260046396]
  We propose AdaptiveDiffusion to reduce noise prediction steps during the denoising process. Our method can significantly speed up the denoising process while generating identical results to the original process, achieving up to an average 25x speedup.
  arXiv  Detail & Related papers  (2024-10-13T15:19:18Z)
• Efficient Diffusion Model for Image Restoration by Residual Shifting [63.02725947015132]
  This study proposes a novel and efficient diffusion model for image restoration. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Our method achieves superior or comparable performance to current state-of-the-art methods on three classical IR tasks.
  arXiv  Detail & Related papers  (2024-03-12T05:06:07Z)
• Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise [34.65659277870287]
  Research on denoising diffusion models has expanded its application to the field of image restoration. We propose Resfusion, a framework that incorporates the residual term into the diffusion forward process. We show that Resfusion exhibits competitive performance on ISTD dataset, LOL dataset and Raindrop dataset with only five sampling steps.
  arXiv  Detail & Related papers  (2023-11-25T02:09:38Z)
• Gradpaint: Gradient-Guided Inpainting with Diffusion Models [71.47496445507862]
  Denoising Diffusion Probabilistic Models (DDPMs) have recently achieved remarkable results in conditional and unconditional image generation. We present GradPaint, which steers the generation towards a globally coherent image. We generalizes well to diffusion models trained on various datasets, improving upon current state-of-the-art supervised and unsupervised methods.
  arXiv  Detail & Related papers  (2023-09-18T09:36:24Z)
• ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolution [84.73658185158222]
  We propose a diffusion model-based super-resolution method called ACDMSR. Our method adapts the standard diffusion model to perform super-resolution through a deterministic iterative denoising process. Our approach generates more visually realistic counterparts for low-resolution images, emphasizing its effectiveness in practical scenarios.
  arXiv  Detail & Related papers  (2023-07-03T06:49:04Z)
• SVNR: Spatially-variant Noise Removal with Denoising Diffusion [43.2405873681083]
  We present a novel formulation of denoising diffusion that assumes a more realistic, spatially-variant noise model. In experiments we demonstrate the advantages of our approach over a strong diffusion model baseline, as well as over a state-of-the-art single image denoising method.
  arXiv  Detail & Related papers  (2023-06-28T09:32:00Z)
• Simultaneous Image-to-Zero and Zero-to-Noise: Diffusion Models with Analytical Image Attenuation [53.04220377034574]
  We propose incorporating an analytical image attenuation process into the forward diffusion process for high-quality (un)conditioned image generation. Our method represents the forward image-to-noise mapping as simultaneous textitimage-to-zero mapping and textitzero-to-noise mapping. We have conducted experiments on unconditioned image generation, textite.g., CIFAR-10 and CelebA-HQ-256, and image-conditioned downstream tasks such as super-resolution, saliency detection, edge detection, and image inpainting.
  arXiv  Detail & Related papers  (2023-06-23T18:08:00Z)
• Parallel Sampling of Diffusion Models [76.3124029406809]
  Diffusion models are powerful generative models but suffer from slow sampling. We present ParaDiGMS, a novel method to accelerate the sampling of pretrained diffusion models by denoising multiple steps in parallel.
  arXiv  Detail & Related papers  (2023-05-25T17:59:42Z)
• CDPMSR: Conditional Diffusion Probabilistic Models for Single Image Super-Resolution [91.56337748920662]
  Diffusion probabilistic models (DPM) have been widely adopted in image-to-image translation. We propose a simple but non-trivial DPM-based super-resolution post-process framework,i.e., cDPMSR. Our method surpasses prior attempts on both qualitative and quantitative results.
  arXiv  Detail & Related papers  (2023-02-14T15:13:33Z)
• Denoising Diffusion Implicit Models [117.03720513930335]
  We present denoising diffusion implicit models (DDIMs) for iterative implicit probabilistic models with the same training procedure as DDPMs. DDIMs can produce high quality samples $10 times$ to $50 times$ faster in terms of wall-clock time compared to DDPMs.
  arXiv  Detail & Related papers  (2020-10-06T06:15:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.