Robust High-Resolution Video Matting with Temporal Guidance

• URL: http://arxiv.org/abs/2108.11515v1
• Date: Wed, 25 Aug 2021 23:48:15 GMT
• Title: Robust High-Resolution Video Matting with Temporal Guidance
• Authors: Shanchuan Lin, Linjie Yang, Imran Saleemi, Soumyadip Sengupta
• Abstract summary: We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU.
• Score: 14.9739044990367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU. Unlike most existing methods that perform video matting frame-by-frame as independent images, our method uses a recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and matting quality. Furthermore, we propose a novel training strategy that enforces our network on both matting and segmentation objectives. This significantly improves our model's robustness. Our method does not require any auxiliary inputs such as a trimap or a pre-captured background image, so it can be widely applied to existing human matting applications.

Related papers

• VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models [58.464465016269614]
  In this paper, we propose a framework for solving high-definition video inverse problems using latent image diffusion models. Our approach leverages latent-space diffusion models to achieve enhanced video quality and resolution. Unlike previous methods, our approach supports multiple aspect ratios and delivers HD-resolution reconstructions in under 2.5 minutes on a single GPU.
  arXiv  Detail & Related papers  (2024-11-29T08:10:49Z)
• ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler [53.98558445900626]
  Current image-to-video diffusion models, while powerful in generating videos from a single frame, need adaptation for two-frame conditioned generation. We introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
  arXiv  Detail & Related papers  (2024-10-08T03:01:54Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis [51.44526084095757]
  We introduce Fairy, a minimalist yet robust adaptation of image-editing diffusion models, enhancing them for video editing applications. Our approach centers on the concept of anchor-based cross-frame attention, a mechanism that implicitly propagates diffusion features across frames. A comprehensive user study, involving 1000 generated samples, confirms that our approach delivers superior quality, decisively outperforming established methods.
  arXiv  Detail & Related papers  (2023-12-20T01:49:47Z)
• Memory-Augmented Non-Local Attention for Video Super-Resolution [61.55700315062226]
  We propose a novel video super-resolution method that aims at generating high-fidelity high-resolution (HR) videos from low-resolution (LR) ones. Previous methods predominantly leverage temporal neighbor frames to assist the super-resolution of the current frame. In contrast, we devise a cross-frame non-local attention mechanism that allows video super-resolution without frame alignment.
  arXiv  Detail & Related papers  (2021-08-25T05:12:14Z)
• Learning Long-Term Style-Preserving Blind Video Temporal Consistency [6.6908747077585105]
  We propose a postprocessing model, to the transformation applied to videos, in the form of a recurrent neural network. Our model is trained using a Ping Pong procedure and its corresponding loss, recently introduced for GAN video generation. We evaluate our model on the DAVIS and videvo.net datasets and show that our approach offers state-of-the-art results concerning flicker removal.
  arXiv  Detail & Related papers  (2021-03-12T13:54:34Z)
• A Plug-and-play Scheme to Adapt Image Saliency Deep Model for Video Data [54.198279280967185]
  This paper proposes a novel plug-and-play scheme to weakly retrain a pretrained image saliency deep model for video data. Our method is simple yet effective for adapting any off-the-shelf pre-trained image saliency deep model to obtain high-quality video saliency detection.
  arXiv  Detail & Related papers  (2020-08-02T13:23:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.