DiffusionMat: Alpha Matting as Sequential Refinement Learning

• URL: http://arxiv.org/abs/2311.13535v1
• Date: Wed, 22 Nov 2023 17:16:44 GMT
• Title: DiffusionMat: Alpha Matting as Sequential Refinement Learning
• Authors: Yangyang Xu, Shengfeng He, Wenqi Shao, Kwan-Yee K. Wong, Yu Qiao, Ping Luo
• Abstract summary: DiffusionMat is an image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes. A correction module adjusts the output at each denoising step, ensuring that the final result is consistent with the input image's structures. We evaluate our model across several image matting benchmarks, and the results indicate that DiffusionMat consistently outperforms existing methods.
• Score: 87.76572845943929
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we introduce DiffusionMat, a novel image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes. Diverging from conventional methods that utilize trimaps merely as loose guidance for alpha matte prediction, our approach treats image matting as a sequential refinement learning process. This process begins with the addition of noise to trimaps and iteratively denoises them using a pre-trained diffusion model, which incrementally guides the prediction towards a clean alpha matte. The key innovation of our framework is a correction module that adjusts the output at each denoising step, ensuring that the final result is consistent with the input image's structures. We also introduce the Alpha Reliability Propagation, a novel technique designed to maximize the utility of available guidance by selectively enhancing the trimap regions with confident alpha information, thus simplifying the correction task. To train the correction module, we devise specialized loss functions that target the accuracy of the alpha matte's edges and the consistency of its opaque and transparent regions. We evaluate our model across several image matting benchmarks, and the results indicate that DiffusionMat consistently outperforms existing methods. Project page at~\url{https://cnnlstm.github.io/DiffusionMat

Related papers

• Texture, Shape and Order Matter: A New Transformer Design for Sequential DeepFake Detection [57.100891917805086]
  Sequential DeepFake detection is an emerging task that predicts the manipulation sequence in order. This paper describes a new Transformer design, called TSOM, by exploring three perspectives: Texture, Shape, and Order of Manipulations.
  arXiv  Detail & Related papers  (2024-04-22T04:47:52Z)
• Learning Saliency From Fixations [0.9208007322096533]
  We present a novel approach for saliency prediction in images, leveraging parallel decoding in transformers to learn saliency solely from fixation maps. Our approach, named Saliency TRansformer (SalTR), achieves metric scores on par with state-of-the-art approaches on the Salicon and MIT300 benchmarks.
  arXiv  Detail & Related papers  (2023-11-23T16:04:41Z)
• Gradpaint: Gradient-Guided Inpainting with Diffusion Models [71.47496445507862]
  Denoising Diffusion Probabilistic Models (DDPMs) have recently achieved remarkable results in conditional and unconditional image generation. We present GradPaint, which steers the generation towards a globally coherent image. We generalizes well to diffusion models trained on various datasets, improving upon current state-of-the-art supervised and unsupervised methods.
  arXiv  Detail & Related papers  (2023-09-18T09:36:24Z)
• Disentangled Pre-training for Image Matting [74.10407744483526]
  Image matting requires high-quality pixel-level human annotations to support the training of a deep model. We propose a self-supervised pre-training approach that can leverage infinite numbers of data to boost the matting performance.
  arXiv  Detail & Related papers  (2023-04-03T08:16:02Z)
• Markup-to-Image Diffusion Models with Scheduled Sampling [111.30188533324954]
  Building on recent advances in image generation, we present a data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameterize the distribution of data using a sequence of denoising operations. We conduct experiments on four markup datasets: mathematical formulas (La), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES)
  arXiv  Detail & Related papers  (2022-10-11T04:56:12Z)
• Semantic Image Matting [75.21022252141474]
  We show how to obtain better alpha mattes by incorporating into our framework semantic classification of matting regions. Specifically, we consider and learn 20 classes of matting patterns, and propose to extend the conventional trimap to semantic trimap. Experiments on multiple benchmarks show that our method outperforms other methods and has achieved the most competitive state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-04-16T16:21:02Z)
• High-Resolution Deep Image Matting [39.72708676319803]
  HDMatt is a first deep learning based image matting approach for high-resolution inputs. Our proposed method sets new state-of-the-art performance on Adobe Image Matting and AlphaMatting benchmarks.
  arXiv  Detail & Related papers  (2020-09-14T17:53:15Z)
• $F$, $B$, Alpha Matting [0.0]
  We propose a low-cost modification to alpha matting networks to also predict the foreground and background colours. Our method achieves the state of the art performance on the Adobe Composition-1k dataset for alpha matte and composite colour quality.
  arXiv  Detail & Related papers  (2020-03-17T13:27:51Z)
• Background Matting [0.40611352512781856]
  This paper investigates the effects of utilising the background information as well as trimap in the process of alpha calculation. To achieve this goal, a state of the art method, AlphaGan is adopted and modified to process the background information as an extra input channel.
  arXiv  Detail & Related papers  (2020-02-11T14:46:57Z)
• Natural Image Matting via Guided Contextual Attention [18.034160025888056]
  We develop a novel end-to-end approach for natural image matting with a guided contextual attention module. The proposed method can mimic information flow of affinity-based methods and utilize rich features learned by deep neural networks simultaneously. Experiment results on Composition-1k testing set and alphamatting.com benchmark dataset demonstrate that our method outperforms state-of-the-art approaches in natural image matting.
  arXiv  Detail & Related papers  (2020-01-13T05:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.