Fill in the ____ (a Diffusion-based Image Inpainting Pipeline)

• URL: http://arxiv.org/abs/2403.16016v1
• Date: Sun, 24 Mar 2024 05:26:55 GMT
• Title: Fill in the ____ (a Diffusion-based Image Inpainting Pipeline)
• Authors: Eyoel Gebre, Krishna Saxena, Timothy Tran,
• Abstract summary: Inpainting is the process of taking an image and generating lost or intentionally occluded portions. Modern inpainting techniques have shown remarkable ability in generating sensible completions. A critical gap in these existing models will be addressed, focusing on the ability to prompt and control what exactly is generated.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image inpainting is the process of taking an image and generating lost or intentionally occluded portions. Inpainting has countless applications including restoring previously damaged pictures, restoring the quality of images that have been degraded due to compression, and removing unwanted objects/text. Modern inpainting techniques have shown remarkable ability in generating sensible completions for images with mask occlusions. In our paper, an overview of the progress of inpainting techniques will be provided, along with identifying current leading approaches, focusing on their strengths and weaknesses. A critical gap in these existing models will be addressed, focusing on the ability to prompt and control what exactly is generated. We will additionally justify why we think this is the natural next progressive step that inpainting models must take, and provide multiple approaches to implementing this functionality. Finally, we will evaluate the results of our approaches by qualitatively checking whether they generate high-quality images that correctly inpaint regions with the objects that they are instructed to produce.

Related papers

• RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting [63.567363455092234]
  RefFusion is a novel 3D inpainting method based on a multi-scale personalization of an image inpainting diffusion model to the given reference view. Our framework achieves state-of-the-art results for object removal while maintaining high controllability.
  arXiv  Detail & Related papers  (2024-04-16T17:50:02Z)
• BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion [61.90969199199739]
  BrushNet is a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM. BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.
  arXiv  Detail & Related papers  (2024-03-11T17:59:31Z)
• GRIG: Few-Shot Generative Residual Image Inpainting [27.252855062283825]
  We present a novel few-shot generative residual image inpainting method that produces high-quality inpainting results. The core idea is to propose an iterative residual reasoning method that incorporates Convolutional Neural Networks (CNNs) for feature extraction. We also propose a novel forgery-patch adversarial training strategy to create faithful textures and detailed appearances.
  arXiv  Detail & Related papers  (2023-04-24T12:19:06Z)
• Perceptual Artifacts Localization for Inpainting [60.5659086595901]
  We propose a new learning task of automatic segmentation of inpainting perceptual artifacts. We train advanced segmentation networks on a dataset to reliably localize inpainting artifacts within inpainted images. We also propose a new evaluation metric called Perceptual Artifact Ratio (PAR), which is the ratio of objectionable inpainted regions to the entire inpainted area.
  arXiv  Detail & Related papers  (2022-08-05T18:50:51Z)
• Cylin-Painting: Seamless {360\textdegree} Panoramic Image Outpainting and Beyond [136.18504104345453]
  We present a Cylin-Painting framework that involves meaningful collaborations between inpainting and outpainting. The proposed algorithm can be effectively extended to other panoramic vision tasks, such as object detection, depth estimation, and image super-resolution.
  arXiv  Detail & Related papers  (2022-04-18T21:18:49Z)
• Deep Two-Stage High-Resolution Image Inpainting [0.0]
  In this article, we propose a method that solves the problem of inpainting arbitrary-size images. For this, we propose to use information from neighboring pixels by shifting the original image in four directions. This approach can work with existing inpainting models, making them almost resolution independent without the need for retraining.
  arXiv  Detail & Related papers  (2021-04-27T20:32:21Z)
• In&Out : Diverse Image Outpainting via GAN Inversion [89.84841983778672]
  Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image.
  arXiv  Detail & Related papers  (2021-04-01T17:59:10Z)
• High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling [122.06593036862611]
  Existing image inpainting methods often produce artifacts when dealing with large holes in real applications. We propose an iterative inpainting method with a feedback mechanism. Experiments show that our method significantly outperforms existing methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2020-05-24T13:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.