FaithFill: Faithful Inpainting for Object Completion Using a Single Reference Image

• URL: http://arxiv.org/abs/2406.07865v1
• Date: Wed, 12 Jun 2024 04:45:33 GMT
• Title: FaithFill: Faithful Inpainting for Object Completion Using a Single Reference Image
• Authors: Rupayan Mallick, Amr Abdalla, Sarah Adel Bargal,
• Abstract summary: FaithFill is a diffusion-based inpainting approach for realistic generation of missing object parts. We demonstrate that FaithFill produces faithful generation of the object's missing parts, together with background/scene preservation, from a single reference image.
• Score: 6.742568054626032
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present FaithFill, a diffusion-based inpainting object completion approach for realistic generation of missing object parts. Typically, multiple reference images are needed to achieve such realistic generation, otherwise the generation would not faithfully preserve shape, texture, color, and background. In this work, we propose a pipeline that utilizes only a single input reference image -having varying lighting, background, object pose, and/or viewpoint. The singular reference image is used to generate multiple views of the object to be inpainted. We demonstrate that FaithFill produces faithful generation of the object's missing parts, together with background/scene preservation, from a single reference image. This is demonstrated through standard similarity metrics, human judgement, and GPT evaluation. Our results are presented on the DreamBooth dataset, and a novel proposed dataset.

Related papers

• RealFill: Reference-Driven Generation for Authentic Image Completion [84.98377627001443]
  RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene.
  arXiv  Detail & Related papers  (2023-09-28T17:59:29Z)
• DreamCom: Finetuning Text-guided Inpainting Model for Image Composition [24.411003826961686]
  We propose DreamCom by treating image composition as text-guided image inpainting customized for certain object. Specifically, we finetune pretrained text-guided image inpainting model based on a few reference images containing the same object. In practice, the inserted object may be adversely affected by the background, so we propose masked attention mechanisms to avoid negative background interference.
  arXiv  Detail & Related papers  (2023-09-27T09:23:50Z)
• AnyDoor: Zero-shot Object-level Image Customization [63.44307304097742]
  This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations. Our model is trained only once and effortlessly generalizes to diverse object-scene combinations at the inference stage.
  arXiv  Detail & Related papers  (2023-07-18T17:59:02Z)
• SIEDOB: Semantic Image Editing by Disentangling Object and Background [5.149242555705579]
  We propose a novel paradigm for semantic image editing. textbfSIEDOB, the core idea of which is to explicitly leverage several heterogeneousworks for objects and backgrounds. We conduct extensive experiments on Cityscapes and ADE20K-Room datasets and exhibit that our method remarkably outperforms the baselines.
  arXiv  Detail & Related papers  (2023-03-23T06:17:23Z)
• Affordance Diffusion: Synthesizing Hand-Object Interactions [81.98499943996394]
  Given an RGB image of an object, we aim to hallucinate plausible images of a human hand interacting with it. We propose a two-step generative approach: a LayoutNet that samples an articulation-agnostic hand-object-interaction layout, and a ContentNet that synthesizes images of a hand grasping the object.
  arXiv  Detail & Related papers  (2023-03-21T17:59:10Z)
• RealFusion: 360{\deg} Reconstruction of Any Object from a Single Image [98.46318529630109]
  We consider the problem of reconstructing a full 360deg photographic model of an object from a single image. We take an off-the-self conditional image generator based on diffusion and engineer a prompt that encourages it to "dream up" novel views of the object.
  arXiv  Detail & Related papers  (2023-02-21T13:25:35Z)
• Foreground-Background Separation through Concept Distillation from Generative Image Foundation Models [6.408114351192012]
  We present a novel method that enables the generation of general foreground-background segmentation models from simple textual descriptions. We show results on the task of segmenting four different objects (humans, dogs, cars, birds) and a use case scenario in medical image analysis.
  arXiv  Detail & Related papers  (2022-12-29T13:51:54Z)
• Shape-guided Object Inpainting [84.18768707298105]
  This work studies a new image inpainting task, i.e. shape-guided object inpainting. We propose a new data preparation method and a novel Contextual Object Generator (CogNet) for the object inpainting task. Experiments demonstrate that the proposed method can generate realistic objects that fit the context in terms of both visual appearance and semantic meanings.
  arXiv  Detail & Related papers  (2022-04-16T17:19:11Z)
• ShaRF: Shape-conditioned Radiance Fields from a Single View [54.39347002226309]
  We present a method for estimating neural scenes representations of objects given only a single image. The core of our method is the estimation of a geometric scaffold for the object. We demonstrate in several experiments the effectiveness of our approach in both synthetic and real images.
  arXiv  Detail & Related papers  (2021-02-17T16:40:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.