Image Harmonization with Region-wise Contrastive Learning

• URL: http://arxiv.org/abs/2205.14058v1
• Date: Fri, 27 May 2022 15:46:55 GMT
• Title: Image Harmonization with Region-wise Contrastive Learning
• Authors: Jingtang Liang and Chi-Man Pun
• Abstract summary: We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
• Score: 51.309905690367835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image harmonization task aims at harmonizing different composite foreground regions according to specific background image. Previous methods would rather focus on improving the reconstruction ability of the generator by some internal enhancements such as attention, adaptive normalization and light adjustment, $etc.$. However, they pay less attention to discriminating the foreground and background appearance features within a restricted generator, which becomes a new challenge in image harmonization task. In this paper, we propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. For the external style fusion, we leverage the external background appearance from the encoder as the style reference to generate harmonized foreground in the decoder. This approach enhances the harmonization ability of the decoder by external background guidance. Moreover, for the contrastive learning scheme, we design a region-wise contrastive loss function for image harmonization task. Specifically, we first introduce a straight-forward samples generation method that selects negative samples from the output harmonized foreground region and selects positive samples from the ground-truth background region. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles, which desirably makes our harmonization network more robust to discriminate the foreground and background style features when harmonizing composite images. Extensive experiments on the benchmark datasets show that our method can achieve a clear improvement in harmonization quality and demonstrate the good generalization capability in real-scenario applications.

Related papers

• FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior [50.0535198082903]
  We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. We showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition.
  arXiv  Detail & Related papers  (2024-07-06T03:35:43Z)
• Intrinsic Harmonization for Illumination-Aware Compositing [0.7366405857677227]
  We introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a re-shading that aligns with the background scene.
  arXiv  Detail & Related papers  (2023-12-06T18:59:03Z)
• FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model [19.170302996189335]
  Our FreePIH method tames the denoising process as a plug-in module for foreground image style transfer. We make use of multi-scale features to enforce the consistency of the content and stability of the foreground objects in the latent space. Our method can surpass representative baselines by large margins.
  arXiv  Detail & Related papers  (2023-11-25T04:23:49Z)
• Learning Global-aware Kernel for Image Harmonization [55.614444363743765]
  Image harmonization aims to solve the visual inconsistency problem in composited images by adaptively adjusting the foreground pixels with the background as references. Existing methods employ local color transformation or region matching between foreground and background, which neglects powerful proximity prior and independently distinguishes fore-/back-ground as a whole part for harmonization. We propose a novel Global-aware Kernel Network (GKNet) to harmonize local regions with comprehensive consideration of long-distance background references.
  arXiv  Detail & Related papers  (2023-05-19T13:49:02Z)
• SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization [29.600429707123645]
  We propose a self-consistent style contrastive learning scheme (SCS-Co) for image harmonization. By dynamically generating multiple negative samples, our SCS-Co can learn more distortion knowledge and well regularize the generated harmonized image. In addition, we propose a background-attentional adaptive instance normalization (BAIN) to achieve an attention-weighted background feature distribution.
  arXiv  Detail & Related papers  (2022-04-29T09:22:01Z)
• Interactive Portrait Harmonization [99.15331091722231]
  Current image harmonization methods consider the entire background as the guidance for harmonization. A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed. Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region.
  arXiv  Detail & Related papers  (2022-03-15T19:30:34Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators? [100.60938767993088]
  We propose a lightweight optimization-based algorithm which could adapt to arbitrary input images and render natural translation effects under flexible objectives. We verify the performance of the proposed framework in facial attribute editing on high-resolution images, where both photo-realism and consistency are required.
  arXiv  Detail & Related papers  (2020-11-19T07:37:31Z)
• BargainNet: Background-Guided Domain Translation for Image Harmonization [26.370523451625466]
  Unharmonious foreground and background downgrade the quality of composite image. Image harmonization, which adjusts the foreground to improve the consistency, is an essential yet challenging task. We propose an image harmonization network with a novel domain code extractor and well-tailored triplet losses.
  arXiv  Detail & Related papers  (2020-09-19T05:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.