Bridging Composite and Real: Towards End-to-end Deep Image Matting

• URL: http://arxiv.org/abs/2010.16188v3
• Date: Wed, 27 Oct 2021 03:31:34 GMT
• Title: Bridging Composite and Real: Towards End-to-end Deep Image Matting
• Authors: Jizhizi Li, Jing Zhang, Stephen J. Maybank, Dacheng Tao
• Abstract summary: We study the roles of semantics and details for image matting. We propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods.
• Score: 88.79857806542006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Extracting accurate foregrounds from natural images benefits many downstream applications such as film production and augmented reality. However, the furry characteristics and various appearance of the foregrounds, e.g., animal and portrait, challenge existing matting methods, which usually require extra user inputs such as trimap or scribbles. To resolve these problems, we study the distinct roles of semantics and details for image matting and decompose the task into two parallel sub-tasks: high-level semantic segmentation and low-level details matting. Specifically, we propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders to learn both tasks in a collaborative manner for end-to-end natural image matting. Besides, due to the limitation of available natural images in the matting task, previous methods typically adopt composite images for training and evaluation, which result in limited generalization ability on real-world images. In this paper, we investigate the domain gap issue between composite images and real-world images systematically by conducting comprehensive analyses of various discrepancies between the foreground and background images. We find that a carefully designed composition route RSSN that aims to reduce the discrepancies can lead to a better model with remarkable generalization ability. Furthermore, we provide a benchmark containing 2,000 high-resolution real-world animal images and 10,000 portrait images along with their manually labeled alpha mattes to serve as a test bed for evaluating matting model's generalization ability on real-world images. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods and effectively reduces the generalization error. The code and the datasets will be released at https://github.com/JizhiziLi/GFM.

Related papers

• ENTED: Enhanced Neural Texture Extraction and Distribution for Reference-based Blind Face Restoration [51.205673783866146]
  We present ENTED, a new framework for blind face restoration that aims to restore high-quality and realistic portrait images. We utilize a texture extraction and distribution framework to transfer high-quality texture features between the degraded input and reference image. The StyleGAN-like architecture in our framework requires high-quality latent codes to generate realistic images.
  arXiv  Detail & Related papers  (2024-01-13T04:54:59Z)
• Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks [50.822601495422916]
  We propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks. Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data. In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
  arXiv  Detail & Related papers  (2024-01-01T14:14:35Z)
• Deep Image Matting: A Comprehensive Survey [85.77905619102802]
  This paper presents a review of recent advancements in image matting in the era of deep learning. We focus on two fundamental sub-tasks: auxiliary input-based image matting and automatic image matting. We discuss relevant applications of image matting and highlight existing challenges and potential opportunities for future research.
  arXiv  Detail & Related papers  (2023-04-10T15:48:55Z)
• Neural Congealing: Aligning Images to a Joint Semantic Atlas [14.348512536556413]
  We present a zero-shot self-supervised framework for aligning semantically-common content across a set of images. Our approach harnesses the power of pre-trained DINO-ViT features to learn. We show that our method performs favorably compared to a state-of-the-art method that requires extensive training on large-scale datasets.
  arXiv  Detail & Related papers  (2023-02-08T09:26:22Z)
• Deep Image Compositing [93.75358242750752]
  We propose a new method which can automatically generate high-quality image composites without any user input. Inspired by Laplacian pyramid blending, a dense-connected multi-stream fusion network is proposed to effectively fuse the information from the foreground and background images. Experiments show that the proposed method can automatically generate high-quality composites and outperforms existing methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-11-04T06:12:24Z)
• High-Resolution Deep Image Matting [39.72708676319803]
  HDMatt is a first deep learning based image matting approach for high-resolution inputs. Our proposed method sets new state-of-the-art performance on Adobe Image Matting and AlphaMatting benchmarks.
  arXiv  Detail & Related papers  (2020-09-14T17:53:15Z)
• Unifying Specialist Image Embedding into Universal Image Embedding [84.0039266370785]
  It is desirable to have a universal deep embedding model applicable to various domains of images. We propose to distill the knowledge in multiple specialists into a universal embedding to solve this problem.
  arXiv  Detail & Related papers  (2020-03-08T02:51:11Z)
• AlphaNet: An Attention Guided Deep Network for Automatic Image Matting [0.0]
  We propose an end to end solution for image matting i.e. high-precision extraction of foreground objects from natural images. We propose a method that assimilates semantic segmentation and deep image matting processes into a single network to generate semantic mattes. We also construct a fashion e-commerce focused dataset with high-quality alpha mattes to facilitate the training and evaluation for image matting.
  arXiv  Detail & Related papers  (2020-03-07T17:25:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.