A Generative Adversarial Framework for Optimizing Image Matting and Harmonization Simultaneously

• URL: http://arxiv.org/abs/2108.06087v1
• Date: Fri, 13 Aug 2021 06:48:14 GMT
• Title: A Generative Adversarial Framework for Optimizing Image Matting and Harmonization Simultaneously
• Authors: Xuqian Ren, Yifan Liu, Chunlei Song
• Abstract summary: We propose a new Generative Adversarial (GAN) framework which optimizing the matting network and the harmonization network based on a self-attention discriminator. Our dataset and dataset generating pipeline can be found in urlhttps://git.io/HaMaGAN
• Score: 7.541357996797061
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Image matting and image harmonization are two important tasks in image composition. Image matting, aiming to achieve foreground boundary details, and image harmonization, aiming to make the background compatible with the foreground, are both promising yet challenging tasks. Previous works consider optimizing these two tasks separately, which may lead to a sub-optimal solution. We propose to optimize matting and harmonization simultaneously to get better performance on both the two tasks and achieve more natural results. We propose a new Generative Adversarial (GAN) framework which optimizing the matting network and the harmonization network based on a self-attention discriminator. The discriminator is required to distinguish the natural images from different types of fake synthesis images. Extensive experiments on our constructed dataset demonstrate the effectiveness of our proposed method. Our dataset and dataset generating pipeline can be found in \url{https://git.io/HaMaGAN}

Related papers

• DiffHarmony: Latent Diffusion Model Meets Image Harmonization [11.500358677234939]
  Diffusion models have promoted the rapid development of image-to-image translation tasks. Fine-tuning pre-trained latent diffusion models from scratch is computationally intensive. In this paper, we adapt a pre-trained latent diffusion model to the image harmonization task to generate harmonious but potentially blurry initial images.
  arXiv  Detail & Related papers  (2024-04-09T09:05:23Z)
• Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks [50.822601495422916]
  We propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks. Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data. In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
  arXiv  Detail & Related papers  (2024-01-01T14:14:35Z)
• Deep Image Harmonization with Learnable Augmentation [17.690945824240348]
  Learnable augmentation is proposed to enrich the illumination diversity of small-scale datasets for better harmonization performance. SycoNet takes in a real image with foreground mask and a random vector to learn suitable color transformation, which is applied to the foreground of this real image to produce a synthetic composite image.
  arXiv  Detail & Related papers  (2023-08-01T08:40:23Z)
• Painterly Image Harmonization in Dual Domains [13.067850524730698]
  We propose a novel painterly harmonization network consisting of a dual-domain generator and a dual-domain discriminator. The dual-domain generator performs harmonization by using AdaIN modules in the spatial domain and our proposed ResFFT modules in the frequency domain. The dual-domain discriminator attempts to distinguish the inharmonious patches based on the spatial feature and frequency feature of each patch, which can enhance the ability of generator in an adversarial manner.
  arXiv  Detail & Related papers  (2022-12-17T11:00:34Z)
• Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
  We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
  arXiv  Detail & Related papers  (2022-05-27T15:46:55Z)
• Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization [59.19214040221055]
  We propose a novel spatial-separated curve rendering network (S$2$CRNet) for efficient and high-resolution image harmonization. The proposed method reduces more than 90% parameters compared with previous methods. Our method can work smoothly on higher resolution images in real-time which is more than 10$times$ faster than the existing methods.
  arXiv  Detail & Related papers  (2021-09-13T07:20:16Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• Adversarial Image Composition with Auxiliary Illumination [53.89445873577062]
  We propose an Adversarial Image Composition Net (AIC-Net) that achieves realistic image composition. A novel branched generation mechanism is proposed, which disentangles the generation of shadows and the transfer of foreground styles. Experiments on pedestrian and car composition tasks show that the proposed AIC-Net achieves superior composition performance.
  arXiv  Detail & Related papers  (2020-09-17T12:58:16Z)
• A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding [57.1077544780653]
  We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
  arXiv  Detail & Related papers  (2020-06-26T08:34:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.