Image Harmonization by Matching Regional References

• URL: http://arxiv.org/abs/2204.04715v1
• Date: Sun, 10 Apr 2022 16:23:06 GMT
• Title: Image Harmonization by Matching Regional References
• Authors: Ziyue Zhu, Zhao Zhang, Zheng Lin, Ruiqi Wu, Zhi Chai, Chun-Le Guo
• Abstract summary: Recent image harmonization methods typically summarize the appearance pattern of global background and apply it to the global foreground without location discrepancy. For a real image, the appearances (illumination, color temperature, saturation, hue, texture, etc) of different regions can vary significantly. Previous methods, which transfer the appearance globally, are not optimal.
• Score: 10.249228010611617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To achieve visual consistency in composite images, recent image harmonization methods typically summarize the appearance pattern of global background and apply it to the global foreground without location discrepancy. However, for a real image, the appearances (illumination, color temperature, saturation, hue, texture, etc) of different regions can vary significantly. So previous methods, which transfer the appearance globally, are not optimal. Trying to solve this issue, we firstly match the contents between the foreground and background and then adaptively adjust every foreground location according to the appearance of its content-related background regions. Further, we design a residual reconstruction strategy, that uses the predicted residual to adjust the appearance, and the composite foreground to reserve the image details. Extensive experiments demonstrate the effectiveness of our method. The source code will be available publicly.

Related papers

• Segment Anything Model Meets Image Harmonization [13.415810438244788]
  Image harmonization is a crucial technique in image composition that aims to seamlessly match the background by adjusting the foreground of composite images. Current methods adopt either global-level or pixel-level feature matching. We propose Semantic-guided Region-aware Instance Normalization (SRIN) that can utilize the semantic segmentation maps output by a pre-trained Segment Anything Model (SAM) to guide the visual consistency learning of foreground and background features.
  arXiv  Detail & Related papers  (2023-12-20T02:57:21Z)
• Intrinsic Harmonization for Illumination-Aware Compositing [0.7366405857677227]
  We introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a re-shading that aligns with the background scene.
  arXiv  Detail & Related papers  (2023-12-06T18:59:03Z)
• Learning Global-aware Kernel for Image Harmonization [55.614444363743765]
  Image harmonization aims to solve the visual inconsistency problem in composited images by adaptively adjusting the foreground pixels with the background as references. Existing methods employ local color transformation or region matching between foreground and background, which neglects powerful proximity prior and independently distinguishes fore-/back-ground as a whole part for harmonization. We propose a novel Global-aware Kernel Network (GKNet) to harmonize local regions with comprehensive consideration of long-distance background references.
  arXiv  Detail & Related papers  (2023-05-19T13:49:02Z)
• Semi-supervised Parametric Real-world Image Harmonization [38.66826236220168]
  Learning-based image harmonization techniques are usually trained to undo synthetic random global transformations applied to a masked foreground in a single ground truth photo. This simulated data does not model many of the important appearance mismatches between foreground and background in real composites. We propose a new semi-supervised training strategy that addresses this problem and lets us learn complex local appearance harmonization from unpaired real composites.
  arXiv  Detail & Related papers  (2023-03-01T01:09:01Z)
• Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
  We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
  arXiv  Detail & Related papers  (2022-05-27T15:46:55Z)
• FRIH: Fine-grained Region-aware Image Harmonization [49.420765789360836]
  We propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH) Our algorithm achieves the best performance on iHarmony4 dataset (PSNR is 38.19 dB) with a lightweight model.
  arXiv  Detail & Related papers  (2022-05-13T04:50:26Z)
• Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN [88.62422914645066]
  We present an algorithm for re-rendering a person from a single image under arbitrary poses. Existing methods often have difficulties in hallucinating occluded contents photo-realistically while preserving the identity and fine details in the source image. We show that our method compares favorably against the state-of-the-art algorithms in both quantitative evaluation and visual comparison.
  arXiv  Detail & Related papers  (2021-09-13T17:59:33Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• Region-aware Adaptive Instance Normalization for Image Harmonization [14.77918186672189]
  To acquire photo-realistic composite images, one must adjust the appearance and visual style of the foreground to be compatible with the background. Existing deep learning methods for harmonizing composite images directly learn an image mapping network from the composite to the real one. We propose a Region-aware Adaptive Instance Normalization (RAIN) module, which explicitly formulates the visual style from the background and adaptively applies them to the foreground.
  arXiv  Detail & Related papers  (2021-06-05T09:57:17Z)
• BargainNet: Background-Guided Domain Translation for Image Harmonization [26.370523451625466]
  Unharmonious foreground and background downgrade the quality of composite image. Image harmonization, which adjusts the foreground to improve the consistency, is an essential yet challenging task. We propose an image harmonization network with a novel domain code extractor and well-tailored triplet losses.
  arXiv  Detail & Related papers  (2020-09-19T05:14:08Z)
• Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [54.836331922449666]
  We propose a Semantic Guidance and Evaluation Network (SGE-Net) to update the structural priors and the inpainted image. It utilizes semantic segmentation map as guidance in each scale of inpainting, under which location-dependent inferences are re-evaluated. Experiments on real-world images of mixed scenes demonstrated the superiority of our proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-15T17:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.