Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand

• URL: http://arxiv.org/abs/2208.03382v1
• Date: Fri, 5 Aug 2022 20:42:13 GMT
• Title: Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand
• Authors: Jitesh Jain, Yuqian Zhou, Ning Yu, Humphrey Shi
• Abstract summary: We claim that the performance of inpainting algorithms can be better judged by the generated structures and textures. In this paper, we propose a novel inpainting network combining the advantages of the two designs. Our model achieves a remarkable visual quality to match state-of-the-art performance in both structure generation and repeating texture synthesis.
• Score: 28.32208483559088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep image inpainting has made impressive progress with recent advances in image generation and processing algorithms. We claim that the performance of inpainting algorithms can be better judged by the generated structures and textures. Structures refer to the generated object boundary or novel geometric structures within the hole, while texture refers to high-frequency details, especially man-made repeating patterns filled inside the structural regions. We believe that better structures are usually obtained from a coarse-to-fine GAN-based generator network while repeating patterns nowadays can be better modeled using state-of-the-art high-frequency fast fourier convolutional layers. In this paper, we propose a novel inpainting network combining the advantages of the two designs. Therefore, our model achieves a remarkable visual quality to match state-of-the-art performance in both structure generation and repeating texture synthesis using a single network. Extensive experiments demonstrate the effectiveness of the method, and our conclusions further highlight the two critical factors of image inpainting quality, structures, and textures, as the future design directions of inpainting networks.

Related papers

• Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects [54.80813150893719]
  We introduce Meta 3D TextureGen: a new feedforward method comprised of two sequential networks aimed at generating high-quality textures in less than 20 seconds. Our method state-of-the-art results in quality and speed by conditioning a text-to-image model on 3D semantics in 2D space and fusing them into a complete and high-resolution UV texture map. In addition, we introduce a texture enhancement network that is capable of up-scaling any texture by an arbitrary ratio, producing 4k pixel resolution textures.
  arXiv  Detail & Related papers  (2024-07-02T17:04:34Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• Progressive with Purpose: Guiding Progressive Inpainting DNNs through Context and Structure [0.0]
  We propose a novel inpainting network that maintains the structural and contextual integrity of a processed image. Inspired by the Gaussian and Laplacian pyramids, the core of the proposed network is a feature extraction module named GLE. Our benchmarking experiments demonstrate that the proposed method achieves clear improvement in performance over many state-of-the-art inpainting algorithms.
  arXiv  Detail & Related papers  (2022-09-21T02:15:02Z)
• Image Inpainting via Conditional Texture and Structure Dual Generation [26.97159780261334]
  We propose a novel two-stream network for image inpainting, which models the structure-constrained texture synthesis and texture-guided structure reconstruction. To enhance the global consistency, a Bi-directional Gated Feature Fusion (Bi-GFF) module is designed to exchange and combine the structure and texture information. Experiments on the CelebA, Paris StreetView and Places2 datasets demonstrate the superiority of the proposed method.
  arXiv  Detail & Related papers  (2021-08-22T15:44:37Z)
• Structure First Detail Next: Image Inpainting with Pyramid Generator [26.94101909283021]
  We propose to build a Pyramid Generator by stacking several sub-generators. Lower-layer sub-generators focus on restoring image structures while the higher-layer sub-generators emphasize image details. Our approach has a learning scheme of progressively increasing hole size, which allows it to restore large-hole images.
  arXiv  Detail & Related papers  (2021-06-16T16:00:16Z)
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [74.29384873537587]
  We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images.
  arXiv  Detail & Related papers  (2021-03-18T05:10:49Z)
• Texture Transform Attention for Realistic Image Inpainting [6.275013056564918]
  We propose a Texture Transform Attention network that better produces the missing region inpainting with fine details. Texture Transform Attention is used to create a new reassembled texture map using fine textures and coarse semantics. We evaluate our model end-to-end with the publicly available datasets CelebA-HQ and Places2.
  arXiv  Detail & Related papers  (2020-12-08T06:28:51Z)
• Retinal Image Segmentation with a Structure-Texture Demixing Network [62.69128827622726]
  The complex structure and texture information are mixed in a retinal image, and distinguishing the information is difficult. Existing methods handle texture and structure jointly, which may lead biased models toward recognizing textures and thus results in inferior segmentation performance. We propose a segmentation strategy that seeks to separate structure and texture components and significantly improve the performance.
  arXiv  Detail & Related papers  (2020-07-15T12:19:03Z)
• Region-adaptive Texture Enhancement for Detailed Person Image Synthesis [86.69934638569815]
  RATE-Net is a novel framework for synthesizing person images with sharp texture details. The proposed framework leverages an additional texture enhancing module to extract appearance information from the source image. Experiments conducted on DeepFashion benchmark dataset have demonstrated the superiority of our framework compared with existing networks.
  arXiv  Detail & Related papers  (2020-05-26T02:33:21Z)
• Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression [113.30411004622508]
  We show that a universal and collaborative visual information representation can be achieved in a hierarchical way. Based on the strong generative capability of deep neural networks, the gap between the base feature layer and enhancement layer is further filled with the feature level texture reconstruction. To improve the efficiency of the proposed framework, the base layer neural network is trained in a multi-task manner.
  arXiv  Detail & Related papers  (2020-04-21T14:32:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.