Self-supervised Matting-specific Portrait Enhancement and Generation

• URL: http://arxiv.org/abs/2208.06601v1
• Date: Sat, 13 Aug 2022 09:00:02 GMT
• Title: Self-supervised Matting-specific Portrait Enhancement and Generation
• Authors: Yangyang Xu Zeyang Zhou and Shengfeng He
• Abstract summary: We use StyleGAN to explore the latent space of GAN models. We optimize multi-scale latent vectors in the latent spaces under four tailored losses. We show that the proposed method can refine real portrait images for arbitrary matting models.
• Score: 40.444011984347505
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We resolve the ill-posed alpha matting problem from a completely different perspective. Given an input portrait image, instead of estimating the corresponding alpha matte, we focus on the other end, to subtly enhance this input so that the alpha matte can be easily estimated by any existing matting models. This is accomplished by exploring the latent space of GAN models. It is demonstrated that interpretable directions can be found in the latent space and they correspond to semantic image transformations. We further explore this property in alpha matting. Particularly, we invert an input portrait into the latent code of StyleGAN, and our aim is to discover whether there is an enhanced version in the latent space which is more compatible with a reference matting model. We optimize multi-scale latent vectors in the latent spaces under four tailored losses, ensuring matting-specificity and subtle modifications on the portrait. We demonstrate that the proposed method can refine real portrait images for arbitrary matting models, boosting the performance of automatic alpha matting by a large margin. In addition, we leverage the generative property of StyleGAN, and propose to generate enhanced portrait data which can be treated as the pseudo GT. It addresses the problem of expensive alpha matte annotation, further augmenting the matting performance of existing models. Code is available at~\url{https://github.com/cnnlstm/StyleGAN_Matting}.

Related papers

• Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective [52.778766190479374]
  Latent-based image generative models have achieved notable success in image generation tasks. Despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. We propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling.
  arXiv  Detail & Related papers  (2024-10-16T12:13:17Z)
• Training Matting Models without Alpha Labels [22.249204770416927]
  This work explores using rough annotations such as trimaps coarsely indicating the foreground/background as supervision. We present that the cooperation between learned semantics from indicated known regions and proper assumed matting rules can help infer alpha values at transition areas. Experiments on AM-2K and P3M-10K dataset show that our paradigm achieves comparable performance with the fine-label-supervised baseline.
  arXiv  Detail & Related papers  (2024-08-20T04:34:06Z)
• DiffusionMat: Alpha Matting as Sequential Refinement Learning [87.76572845943929]
  DiffusionMat is an image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes. A correction module adjusts the output at each denoising step, ensuring that the final result is consistent with the input image's structures. We evaluate our model across several image matting benchmarks, and the results indicate that DiffusionMat consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2023-11-22T17:16:44Z)
• Adversarially-Guided Portrait Matting [0.0]
  We present a method for generating alpha mattes using a limited data source. We pretrain a novel transformerbased model (StyleMatte) on portrait datasets. We utilize this model to provide image-mask pairs for the StyleGAN3-based network (StyleMatteGAN)
  arXiv  Detail & Related papers  (2023-05-04T16:45:04Z)
• High-Resolution GAN Inversion for Degraded Images in Large Diverse Datasets [39.21692649763314]
  In this paper, we present a novel GAN inversion framework that utilizes the powerful generative ability of StyleGAN-XL. To ease the inversion challenge with StyleGAN-XL, Clustering & Regularize Inversion (CRI) is proposed. We validate our CRI scheme on multiple restoration tasks (i.e., inpainting, colorization, and super-resolution) of complex natural images, and show preferable quantitative and qualitative results.
  arXiv  Detail & Related papers  (2023-02-07T11:24:11Z)
• Spatial Steerability of GANs via Self-Supervision from Discriminator [123.27117057804732]
  We propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects.
  arXiv  Detail & Related papers  (2023-01-20T07:36:29Z)
• Alpha Matte Generation from Single Input for Portrait Matting [79.62140902232628]
  The goal is to predict an alpha matte that identifies the effect of each pixel on the foreground subject. Traditional approaches and most of the existing works utilized an additional input, e.g., trimap, background image, to predict alpha matte. We introduce an additional input-free approach to perform portrait matting using Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2021-06-06T18:53:42Z)
• Smart Scribbles for Image Mating [90.18035889903909]
  We propose an interactive framework, referred to as smart scribbles, to guide users to draw few scribbles on the input images. It infers the most informative regions of an image for drawing scribbles to indicate different categories. It then spreads these scribbles to the rest of the image via our well-designed two-phase propagation.
  arXiv  Detail & Related papers  (2021-03-31T13:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.