Reference-Guided Texture and Structure Inference for Image Inpainting

• URL: http://arxiv.org/abs/2207.14498v1
• Date: Fri, 29 Jul 2022 06:26:03 GMT
• Title: Reference-Guided Texture and Structure Inference for Image Inpainting
• Authors: Taorong Liu, Liang Liao, Zheng Wang, Shin'ichi Satoh
• Abstract summary: We build a benchmark dataset containing 10K pairs of input and reference images for reference-guided inpainting. We adopt an encoder-decoder structure to infer the texture and structure features of the input image. A feature alignment module is further designed to refine these features of the input image with the guidance of a reference image.
• Score: 25.775006005766222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing learning-based image inpainting methods are still in challenge when facing complex semantic environments and diverse hole patterns. The prior information learned from the large scale training data is still insufficient for these situations. Reference images captured covering the same scenes share similar texture and structure priors with the corrupted images, which offers new prospects for the image inpainting tasks. Inspired by this, we first build a benchmark dataset containing 10K pairs of input and reference images for reference-guided inpainting. Then we adopt an encoder-decoder structure to separately infer the texture and structure features of the input image considering their pattern discrepancy of texture and structure during inpainting. A feature alignment module is further designed to refine these features of the input image with the guidance of a reference image. Both quantitative and qualitative evaluations demonstrate the superiority of our method over the state-of-the-art methods in terms of completing complex holes.

Related papers

• ENTED: Enhanced Neural Texture Extraction and Distribution for Reference-based Blind Face Restoration [51.205673783866146]
  We present ENTED, a new framework for blind face restoration that aims to restore high-quality and realistic portrait images. We utilize a texture extraction and distribution framework to transfer high-quality texture features between the degraded input and reference image. The StyleGAN-like architecture in our framework requires high-quality latent codes to generate realistic images.
  arXiv  Detail & Related papers  (2024-01-13T04:54:59Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• TransRef: Multi-Scale Reference Embedding Transformer for Reference-Guided Image Inpainting [45.31389892299325]
  We propose a transformer-based encoder-decoder network, named TransRef, for reference-guided image inpainting. For precise utilization of the reference features for guidance, a reference-patch alignment (Ref-PA) module is proposed to align the patch features of the reference and corrupted images. We construct a publicly accessible benchmark dataset containing 50K pairs of input and reference images.
  arXiv  Detail & Related papers  (2023-06-20T13:31:33Z)
• SceneComposer: Any-Level Semantic Image Synthesis [80.55876413285587]
  We propose a new framework for conditional image synthesis from semantic layouts of any precision levels. The framework naturally reduces to text-to-image (T2I) at the lowest level with no shape information, and it becomes segmentation-to-image (S2I) at the highest level. We introduce several novel techniques to address the challenges coming with this new setup.
  arXiv  Detail & Related papers  (2022-11-21T18:59:05Z)
• Delving Globally into Texture and Structure for Image Inpainting [20.954875933730808]
  Image inpainting has achieved remarkable progress and inspired abundant methods, where the critical bottleneck is identified as how to fulfill the high-frequency structure and low-frequency texture information on the masked regions with semantics. In this paper, we delve globally into texture and structure information to well capture the semantics for image inpainting. Our model is tovolution to the fashionable arts, such as Conal Neural Networks (CNNs), Attention and Transformer model, from the perspective of texture and structure information for image inpainting.
  arXiv  Detail & Related papers  (2022-09-17T02:19:26Z)
• Image Inpainting Guided by Coherence Priors of Semantics and Textures [62.92586889409379]
  We introduce coherence priors between the semantics and textures which make it possible to concentrate on completing separate textures in a semantic-wise manner. We also propose two coherence losses to constrain the consistency between the semantics and the inpainted image in terms of the overall structure and detailed textures.
  arXiv  Detail & Related papers  (2020-12-15T02:59:37Z)
• Region-adaptive Texture Enhancement for Detailed Person Image Synthesis [86.69934638569815]
  RATE-Net is a novel framework for synthesizing person images with sharp texture details. The proposed framework leverages an additional texture enhancing module to extract appearance information from the source image. Experiments conducted on DeepFashion benchmark dataset have demonstrated the superiority of our framework compared with existing networks.
  arXiv  Detail & Related papers  (2020-05-26T02:33:21Z)
• Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [54.836331922449666]
  We propose a Semantic Guidance and Evaluation Network (SGE-Net) to update the structural priors and the inpainted image. It utilizes semantic segmentation map as guidance in each scale of inpainting, under which location-dependent inferences are re-evaluated. Experiments on real-world images of mixed scenes demonstrated the superiority of our proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-15T17:49:20Z)
• Learning to Incorporate Structure Knowledge for Image Inpainting [20.93448933499842]
  This paper develops a multi-task learning framework that attempts to incorporate the image structure knowledge to assist image inpainting. The primary idea is to train a shared generator to simultaneously complete the corrupted image and corresponding structures. In the meantime, we also introduce a structure embedding scheme to explicitly embed the learned structure features into the inpainting process.
  arXiv  Detail & Related papers  (2020-02-11T02:22:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.