Multi-feature Co-learning for Image Inpainting

• URL: http://arxiv.org/abs/2205.10578v1
• Date: Sat, 21 May 2022 12:15:26 GMT
• Title: Multi-feature Co-learning for Image Inpainting
• Authors: Jiayu Lin, Yuan-Gen Wang, Wenzhi Tang, Aifeng Li
• Abstract summary: In this paper, we design a deep multi-feature co-learning network for image inpainting. To be specific, we first use two branches to learn structure features and texture features separately. The proposed SDFF module integrates structure features into texture features, and meanwhile uses texture features as an auxiliary in generating structure features.
• Score: 2.4571440831539824
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image inpainting has achieved great advances by simultaneously leveraging image structure and texture features. However, due to lack of effective multi-feature fusion techniques, existing image inpainting methods still show limited improvement. In this paper, we design a deep multi-feature co-learning network for image inpainting, which includes Soft-gating Dual Feature Fusion (SDFF) and Bilateral Propagation Feature Aggregation (BPFA) modules. To be specific, we first use two branches to learn structure features and texture features separately. Then the proposed SDFF module integrates structure features into texture features, and meanwhile uses texture features as an auxiliary in generating structure features. Such a co-learning strategy makes the structure and texture features more consistent. Next, the proposed BPFA module enhances the connection from local feature to overall consistency by co-learning contextual attention, channel-wise information and feature space, which can further refine the generated structures and textures. Finally, extensive experiments are performed on benchmark datasets, including CelebA, Places2, and Paris StreetView. Experimental results demonstrate the superiority of the proposed method over the state-of-the-art. The source codes are available at https://github.com/GZHU-DVL/MFCL-Inpainting.

Related papers

• Selective Vision-Language Subspace Projection for Few-shot CLIP [55.361337202198925]
  We introduce a method called Selective Vision-Language Subspace Projection (SSP) SSP incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks.
  arXiv  Detail & Related papers  (2024-07-24T03:45:35Z)
• Beyond First-Order: A Multi-Scale Approach to Finger Knuckle Print Biometrics [12.218178491042176]
  finger knuckle prints (FKPs) have gained attention due to their rich textural patterns, positioning them as a promising biometric for identity recognition. Prior FKP recognition methods predominantly leverage first-order feature descriptors, which capture intricate texture details but fail to account for structural information. This paper introduces a novel FKP recognition approach, the Dual-Order Texture Competition Network (DOTCNet), designed to capture texture information in FKP images comprehensively.
  arXiv  Detail & Related papers  (2024-06-28T06:06:01Z)
• SSPFusion: A Semantic Structure-Preserving Approach for Infrared and Visible Image Fusion [30.55433673796615]
  Most existing learning-based infrared and visible image fusion (IVIF) methods exhibit massive redundant information in the fusion images. We propose a semantic structure-preserving approach for IVIF, namely SSPFusion. Our method is able to generate high-quality fusion images from pairs of infrared and visible images, which can boost the performance of downstream computer-vision tasks.
  arXiv  Detail & Related papers  (2023-09-26T08:13:32Z)
• LE2Fusion: A novel local edge enhancement module for infrared and visible image fusion [7.11574718614606]
  Under complex illumination conditions, few algorithms pay attention to the edge information of local regions. We propose a fusion network based on the local edge enhancement, named LE2Fusion. Experiments demonstrate that the proposed method exhibits better fusion performance than the state-of-the-art fusion methods on public datasets.
  arXiv  Detail & Related papers  (2023-05-27T05:37:02Z)
• Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance [49.94504248096527]
  We propose a Depth-Guided Outpainting Network (DGONet) to model the feature representations of different modalities. Two components are designed to implement: 1) The Multimodal Learning Module produces unique depth and RGB feature representations from perspectives of different modal characteristics. We specially design an additional constraint strategy consisting of Cross-modal Loss and Edge Loss to enhance ambiguous contours and expedite reliable content generation.
  arXiv  Detail & Related papers  (2022-04-12T06:06:50Z)
• Multi-level Second-order Few-shot Learning [111.0648869396828]
  We propose a Multi-level Second-order (MlSo) few-shot learning network for supervised or unsupervised few-shot image classification and few-shot action recognition. We leverage so-called power-normalized second-order base learner streams combined with features that express multiple levels of visual abstraction. We demonstrate respectable results on standard datasets such as Omniglot, mini-ImageNet, tiered-ImageNet, Open MIC, fine-grained datasets such as CUB Birds, Stanford Dogs and Cars, and action recognition datasets such as HMDB51, UCF101, and mini-MIT.
  arXiv  Detail & Related papers  (2022-01-15T19:49:00Z)
• Image Inpainting via Conditional Texture and Structure Dual Generation [26.97159780261334]
  We propose a novel two-stream network for image inpainting, which models the structure-constrained texture synthesis and texture-guided structure reconstruction. To enhance the global consistency, a Bi-directional Gated Feature Fusion (Bi-GFF) module is designed to exchange and combine the structure and texture information. Experiments on the CelebA, Paris StreetView and Places2 datasets demonstrate the superiority of the proposed method.
  arXiv  Detail & Related papers  (2021-08-22T15:44:37Z)
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [74.29384873537587]
  We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images.
  arXiv  Detail & Related papers  (2021-03-18T05:10:49Z)
• Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression [113.30411004622508]
  We show that a universal and collaborative visual information representation can be achieved in a hierarchical way. Based on the strong generative capability of deep neural networks, the gap between the base feature layer and enhancement layer is further filled with the feature level texture reconstruction. To improve the efficiency of the proposed framework, the base layer neural network is trained in a multi-task manner.
  arXiv  Detail & Related papers  (2020-04-21T14:32:49Z)
• AlignSeg: Feature-Aligned Segmentation Networks [109.94809725745499]
  We propose Feature-Aligned Networks (AlignSeg) to address misalignment issues during the feature aggregation process. Our network achieves new state-of-the-art mIoU scores of 82.6% and 45.95%, respectively.
  arXiv  Detail & Related papers  (2020-02-24T10:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.