RealFill: Reference-Driven Generation for Authentic Image Completion

• URL: http://arxiv.org/abs/2309.16668v2
• Date: Tue, 14 May 2024 17:58:28 GMT
• Title: RealFill: Reference-Driven Generation for Authentic Image Completion
• Authors: Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E. Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, Michael Rubinstein,
• Abstract summary: RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene.
• Score: 84.98377627001443
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in generative imagery have brought forth outpainting and inpainting models that can produce high-quality, plausible image content in unknown regions. However, the content these models hallucinate is necessarily inauthentic, since they are unaware of the true scene. In this work, we propose RealFill, a novel generative approach for image completion that fills in missing regions of an image with the content that should have been there. RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. These reference images do not have to be aligned with the target image, and can be taken with drastically varying viewpoints, lighting conditions, camera apertures, or image styles. Once personalized, RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene. We evaluate RealFill on a new image completion benchmark that covers a set of diverse and challenging scenarios, and find that it outperforms existing approaches by a large margin. Project page: https://realfill.github.io

Related papers

• How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation Threshold [50.33428591760124]
  We study the relationship between a concept's frequency in the training dataset and the ability of a model to imitate it. We propose an efficient approach that estimates the imitation threshold without incurring the colossal cost of training multiple models from scratch.
  arXiv  Detail & Related papers  (2024-10-19T06:28:14Z)
• FaithFill: Faithful Inpainting for Object Completion Using a Single Reference Image [6.742568054626032]
  FaithFill is a diffusion-based inpainting approach for realistic generation of missing object parts. We demonstrate that FaithFill produces faithful generation of the object's missing parts, together with background/scene preservation, from a single reference image.
  arXiv  Detail & Related papers  (2024-06-12T04:45:33Z)
• Text-image guided Diffusion Model for generating Deepfake celebrity interactions [50.37578424163951]
  Diffusion models have recently demonstrated highly realistic visual content generation. This paper devises and explores a novel method in that regard. Our results show that with the devised scheme, it is possible to create fake visual content with alarming realism.
  arXiv  Detail & Related papers  (2023-09-26T08:24:37Z)
• Diverse Semantic Image Editing with Style Codes [6.7737387715834725]
  We propose a framework that can encode visible and partially visible objects with a novel mechanism to achieve consistency in the style encoding and final generations. Our method achieves better quantitative results and also provides diverse results.
  arXiv  Detail & Related papers  (2023-09-25T09:22:18Z)
• LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [121.45667242282721]
  We propose a coarse-to-fine paradigm to achieve layout planning and image generation. Our proposed method outperforms the state-of-the-art models in terms of photorealistic layout and image generation.
  arXiv  Detail & Related papers  (2023-08-09T17:45:04Z)
• Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur [68.24599239479326]
  We develop a hybrid neural rendering model that makes image-based representation and neural 3D representation join forces to render high-quality, view-consistent images. Our model surpasses state-of-the-art point-based methods for novel view synthesis.
  arXiv  Detail & Related papers  (2023-04-25T08:36:33Z)
• Fake it till you make it: Learning transferable representations from synthetic ImageNet clones [30.264601433216246]
  We show that ImageNet clones can close a large part of the gap between models produced by synthetic images and models trained with real images. We show that models trained on synthetic images exhibit strong generalization properties and perform on par with models trained on real data for transfer.
  arXiv  Detail & Related papers  (2022-12-16T11:44:01Z)
• A Shared Representation for Photorealistic Driving Simulators [83.5985178314263]
  We propose to improve the quality of generated images by rethinking the discriminator architecture. The focus is on the class of problems where images are generated given semantic inputs, such as scene segmentation maps or human body poses. We aim to learn a shared latent representation that encodes enough information to jointly do semantic segmentation, content reconstruction, along with a coarse-to-fine grained adversarial reasoning.
  arXiv  Detail & Related papers  (2021-12-09T18:59:21Z)
• Full-Glow: Fully conditional Glow for more realistic image generation [9.30816997952245]
  Full-Glow is a conditional Glow architecture for generating plausible and realistic images of novel street scenes. Benchmark comparisons show our model to outperform recent works in terms of the semantic segmentation performance of a pretrained PSPNet.
  arXiv  Detail & Related papers  (2020-12-10T17:37:43Z)
• Conditional Image Generation and Manipulation for User-Specified Content [6.6081578501076494]
  We propose a single pipeline for text-to-image generation and manipulation. In the first part of our pipeline we introduce textStyleGAN, a model that is conditioned on text. In the second part of our pipeline we make use of the pre-trained weights of textStyleGAN to perform semantic facial image manipulation.
  arXiv  Detail & Related papers  (2020-05-11T08:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.