Deep Image Harmonization with Learnable Augmentation

• URL: http://arxiv.org/abs/2308.00376v1
• Date: Tue, 1 Aug 2023 08:40:23 GMT
• Title: Deep Image Harmonization with Learnable Augmentation
• Authors: Li Niu, Junyan Cao, Wenyan Cong, Liqing Zhang
• Abstract summary: Learnable augmentation is proposed to enrich the illumination diversity of small-scale datasets for better harmonization performance. SycoNet takes in a real image with foreground mask and a random vector to learn suitable color transformation, which is applied to the foreground of this real image to produce a synthetic composite image.
• Score: 17.690945824240348
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The goal of image harmonization is adjusting the foreground appearance in a composite image to make the whole image harmonious. To construct paired training images, existing datasets adopt different ways to adjust the illumination statistics of foregrounds of real images to produce synthetic composite images. However, different datasets have considerable domain gap and the performances on small-scale datasets are limited by insufficient training data. In this work, we explore learnable augmentation to enrich the illumination diversity of small-scale datasets for better harmonization performance. In particular, our designed SYthetic COmposite Network (SycoNet) takes in a real image with foreground mask and a random vector to learn suitable color transformation, which is applied to the foreground of this real image to produce a synthetic composite image. Comprehensive experiments demonstrate the effectiveness of our proposed learnable augmentation for image harmonization. The code of SycoNet is released at https://github.com/bcmi/SycoNet-Adaptive-Image-Harmonization.

Related papers

• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks [50.822601495422916]
  We propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks. Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data. In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
  arXiv  Detail & Related papers  (2024-01-01T14:14:35Z)
• ControlCom: Controllable Image Composition using Diffusion Model [45.48263800282992]
  We propose a controllable image composition method that unifies four tasks in one diffusion model. We also propose a local enhancement module to enhance the foreground details in the diffusion model. The proposed method is evaluated on both public benchmark and real-world data.
  arXiv  Detail & Related papers  (2023-08-19T14:56:44Z)
• Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation [20.302430505018]
  We show that using global information to guide foreground feature transformation could achieve significant improvement. We also propose to transfer the foreground-background relation from real images to composite images, which can provide intermediate supervision for the transformed encoder features.
  arXiv  Detail & Related papers  (2023-08-01T07:53:25Z)
• CcHarmony: Color-checker based Image Harmonization Dataset [13.651034765401386]
  Image harmonization targets at adjusting the foreground in a composite image to make it compatible with the background, producing a more realistic and harmonious image. We construct an image harmonization dataset called ccHarmony, which is named after color checker (cc)
  arXiv  Detail & Related papers  (2022-06-01T23:57:16Z)
• Enhancing Low-Light Images in Real World via Cross-Image Disentanglement [58.754943762945864]
  We propose a new low-light image enhancement dataset consisting of misaligned training images with real-world corruptions. Our model achieves state-of-the-art performances on both the newly proposed dataset and other popular low-light datasets.
  arXiv  Detail & Related papers  (2022-01-10T03:12:52Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• A Generative Adversarial Framework for Optimizing Image Matting and Harmonization Simultaneously [7.541357996797061]
  We propose a new Generative Adversarial (GAN) framework which optimizing the matting network and the harmonization network based on a self-attention discriminator. Our dataset and dataset generating pipeline can be found in urlhttps://git.io/HaMaGAN
  arXiv  Detail & Related papers  (2021-08-13T06:48:14Z)
• Deep Image Harmonization by Bridging the Reality Gap [18.86655082192153]
  We propose to construct a large-scale rendered harmonization dataset RHHarmony with fewer human efforts to augment the existing real-world dataset. To leverage both real-world images and rendered images, we propose a cross-domain harmonization network CharmNet.
  arXiv  Detail & Related papers  (2021-03-31T14:19:56Z)
• Deep Image Compositing [93.75358242750752]
  We propose a new method which can automatically generate high-quality image composites without any user input. Inspired by Laplacian pyramid blending, a dense-connected multi-stream fusion network is proposed to effectively fuse the information from the foreground and background images. Experiments show that the proposed method can automatically generate high-quality composites and outperforms existing methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-11-04T06:12:24Z)
• Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition [67.9464567157846]
  We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-06-29T12:53:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.