Painterly Image Harmonization in Dual Domains

• URL: http://arxiv.org/abs/2212.08846v4
• Date: Tue, 4 Jul 2023 07:31:18 GMT
• Title: Painterly Image Harmonization in Dual Domains
• Authors: Junyan Cao, Yan Hong, Li Niu
• Abstract summary: We propose a novel painterly harmonization network consisting of a dual-domain generator and a dual-domain discriminator. The dual-domain generator performs harmonization by using AdaIN modules in the spatial domain and our proposed ResFFT modules in the frequency domain. The dual-domain discriminator attempts to distinguish the inharmonious patches based on the spatial feature and frequency feature of each patch, which can enhance the ability of generator in an adversarial manner.
• Score: 13.067850524730698
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image harmonization aims to produce visually harmonious composite images by adjusting the foreground appearance to be compatible with the background. When the composite image has photographic foreground and painterly background, the task is called painterly image harmonization. There are only few works on this task, which are either time-consuming or weak in generating well-harmonized results. In this work, we propose a novel painterly harmonization network consisting of a dual-domain generator and a dual-domain discriminator, which harmonizes the composite image in both spatial domain and frequency domain. The dual-domain generator performs harmonization by using AdaIN modules in the spatial domain and our proposed ResFFT modules in the frequency domain. The dual-domain discriminator attempts to distinguish the inharmonious patches based on the spatial feature and frequency feature of each patch, which can enhance the ability of generator in an adversarial manner. Extensive experiments on the benchmark dataset show the effectiveness of our method. Our code and model are available at https://github.com/bcmi/PHDNet-Painterly-Image-Harmonization.

Related papers

• A Dual Domain Multi-exposure Image Fusion Network based on the Spatial-Frequency Integration [57.14745782076976]
  Multi-exposure image fusion aims to generate a single high-dynamic image by integrating images with different exposures. We propose a novelty perspective on multi-exposure image fusion via the Spatial-Frequency Integration Framework, named MEF-SFI. Our method achieves visual-appealing fusion results against state-of-the-art multi-exposure image fusion approaches.
  arXiv  Detail & Related papers  (2023-12-17T04:45:15Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models [89.76587063609806]
  We study the denoising diffusion probabilistic model (DDPM) in wavelet space, instead of pixel space, for visual synthesis. By explicitly modeling the wavelet signals, we find our model is able to generate images with higher quality on several datasets.
  arXiv  Detail & Related papers  (2023-07-27T06:53:16Z)
• BlindHarmony: "Blind" Harmonization for MR Images via Flow model [1.765282368080009]
  In MRI, images of the same contrast from the same subject can exhibit noticeable differences when acquired using different hardware, sequences, or scan parameters. These differences create a domain gap that needs to be bridged by image harmonization. Deep learning-based approaches have been proposed to achieve image harmonization. We propose BlindHarmony, which utilizes only target domain data for training but still has the capability to harmonize images from unseen domains.
  arXiv  Detail & Related papers  (2023-05-18T06:04:24Z)
• Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
  We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
  arXiv  Detail & Related papers  (2022-05-27T15:46:55Z)
• FRIH: Fine-grained Region-aware Image Harmonization [49.420765789360836]
  We propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH) Our algorithm achieves the best performance on iHarmony4 dataset (PSNR is 38.19 dB) with a lightweight model.
  arXiv  Detail & Related papers  (2022-05-13T04:50:26Z)
• Interactive Portrait Harmonization [99.15331091722231]
  Current image harmonization methods consider the entire background as the guidance for harmonization. A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed. Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region.
  arXiv  Detail & Related papers  (2022-03-15T19:30:34Z)
• A Generative Adversarial Framework for Optimizing Image Matting and Harmonization Simultaneously [7.541357996797061]
  We propose a new Generative Adversarial (GAN) framework which optimizing the matting network and the harmonization network based on a self-attention discriminator. Our dataset and dataset generating pipeline can be found in urlhttps://git.io/HaMaGAN
  arXiv  Detail & Related papers  (2021-08-13T06:48:14Z)
• Deep Image Harmonization by Bridging the Reality Gap [18.86655082192153]
  We propose to construct a large-scale rendered harmonization dataset RHHarmony with fewer human efforts to augment the existing real-world dataset. To leverage both real-world images and rendered images, we propose a cross-domain harmonization network CharmNet.
  arXiv  Detail & Related papers  (2021-03-31T14:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.