Region-aware Adaptive Instance Normalization for Image Harmonization
- URL: http://arxiv.org/abs/2106.02853v1
- Date: Sat, 5 Jun 2021 09:57:17 GMT
- Title: Region-aware Adaptive Instance Normalization for Image Harmonization
- Authors: Jun Ling, Han Xue, Li Song, Rong Xie and Xiao Gu
- Abstract summary: To acquire photo-realistic composite images, one must adjust the appearance and visual style of the foreground to be compatible with the background.
Existing deep learning methods for harmonizing composite images directly learn an image mapping network from the composite to the real one.
We propose a Region-aware Adaptive Instance Normalization (RAIN) module, which explicitly formulates the visual style from the background and adaptively applies them to the foreground.
- Score: 14.77918186672189
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image composition plays a common but important role in photo editing. To
acquire photo-realistic composite images, one must adjust the appearance and
visual style of the foreground to be compatible with the background. Existing
deep learning methods for harmonizing composite images directly learn an image
mapping network from the composite to the real one, without explicit
exploration on visual style consistency between the background and the
foreground images. To ensure the visual style consistency between the
foreground and the background, in this paper, we treat image harmonization as a
style transfer problem. In particular, we propose a simple yet effective
Region-aware Adaptive Instance Normalization (RAIN) module, which explicitly
formulates the visual style from the background and adaptively applies them to
the foreground. With our settings, our RAIN module can be used as a drop-in
module for existing image harmonization networks and is able to bring
significant improvements. Extensive experiments on the existing image
harmonization benchmark datasets show the superior capability of the proposed
method. Code is available at {https://github.com/junleen/RainNet}.
Related papers
- Segment Anything Model Meets Image Harmonization [13.415810438244788]
Image harmonization is a crucial technique in image composition that aims to seamlessly match the background by adjusting the foreground of composite images.
Current methods adopt either global-level or pixel-level feature matching.
We propose Semantic-guided Region-aware Instance Normalization (SRIN) that can utilize the semantic segmentation maps output by a pre-trained Segment Anything Model (SAM) to guide the visual consistency learning of foreground and background features.
arXiv Detail & Related papers (2023-12-20T02:57:21Z) - Intrinsic Harmonization for Illumination-Aware Compositing [0.7366405857677227]
We introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain.
First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region.
A network then refines this inferred shading to generate a re-shading that aligns with the background scene.
arXiv Detail & Related papers (2023-12-06T18:59:03Z) - Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme.
Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
arXiv Detail & Related papers (2022-05-27T15:46:55Z) - Retrieval-based Spatially Adaptive Normalization for Semantic Image
Synthesis [68.1281982092765]
We propose a novel normalization module, termed as REtrieval-based Spatially AdaptIve normaLization (RESAIL)
RESAIL provides pixel level fine-grained guidance to the normalization architecture.
Experiments on several challenging datasets show that our RESAIL performs favorably against state-of-the-arts in terms of quantitative metrics, visual quality, and subjective evaluation.
arXiv Detail & Related papers (2022-04-06T14:21:39Z) - Interactive Portrait Harmonization [99.15331091722231]
Current image harmonization methods consider the entire background as the guidance for harmonization.
A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed.
Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region.
arXiv Detail & Related papers (2022-03-15T19:30:34Z) - SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited.
Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
arXiv Detail & Related papers (2021-08-15T19:51:33Z) - Controllable Person Image Synthesis with Spatially-Adaptive Warped
Normalization [72.65828901909708]
Controllable person image generation aims to produce realistic human images with desirable attributes.
We introduce a novel Spatially-Adaptive Warped Normalization (SAWN), which integrates a learned flow-field to warp modulation parameters.
We propose a novel self-training part replacement strategy to refine the pretrained model for the texture-transfer task.
arXiv Detail & Related papers (2021-05-31T07:07:44Z) - BargainNet: Background-Guided Domain Translation for Image Harmonization [26.370523451625466]
Unharmonious foreground and background downgrade the quality of composite image.
Image harmonization, which adjusts the foreground to improve the consistency, is an essential yet challenging task.
We propose an image harmonization network with a novel domain code extractor and well-tailored triplet losses.
arXiv Detail & Related papers (2020-09-19T05:14:08Z) - Foreground-aware Semantic Representations for Image Harmonization [5.156484100374058]
We propose a novel architecture to utilize the space of high-level features learned by a pre-trained classification network.
We extensively evaluate the proposed method on existing image harmonization benchmark and set up a new state-of-the-art in terms of MSE and PSNR metrics.
arXiv Detail & Related papers (2020-06-01T09:27:20Z) - BachGAN: High-Resolution Image Synthesis from Salient Object Layout [78.51640906030244]
We propose a new task towards more practical application for image generation - high-quality image synthesis from salient object layout.
Two main challenges spring from this new task: (i) how to generate fine-grained details and realistic textures without segmentation map input; and (ii) how to create a background and weave it seamlessly into standalone objects.
By generating the hallucinated background representation dynamically, our model can synthesize high-resolution images with both photo-realistic foreground and integral background.
arXiv Detail & Related papers (2020-03-26T00:54:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.