Towards Online Real-Time Memory-based Video Inpainting Transformers

• URL: http://arxiv.org/abs/2403.16161v1
• Date: Sun, 24 Mar 2024 14:02:25 GMT
• Title: Towards Online Real-Time Memory-based Video Inpainting Transformers
• Authors: Guillaume Thiry, Hao Tang, Radu Timofte, Luc Van Gool,
• Abstract summary: Inpainting tasks have seen significant improvements in recent years with the rise of deep neural networks and, in particular, vision transformers. We propose a framework to adapt existing inpainting transformers to constraints by memorizing and refining redundant computations. Using this framework with some of the most recent inpainting models, we show great online results with a consistent throughput above 20 frames per second.
• Score: 95.90235034520167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video inpainting tasks have seen significant improvements in recent years with the rise of deep neural networks and, in particular, vision transformers. Although these models show promising reconstruction quality and temporal consistency, they are still unsuitable for live videos, one of the last steps to make them completely convincing and usable. The main limitations are that these state-of-the-art models inpaint using the whole video (offline processing) and show an insufficient frame rate. In our approach, we propose a framework to adapt existing inpainting transformers to these constraints by memorizing and refining redundant computations while maintaining a decent inpainting quality. Using this framework with some of the most recent inpainting models, we show great online results with a consistent throughput above 20 frames per second. The code and pretrained models will be made available upon acceptance.

Related papers

• Video Diffusion Models are Strong Video Inpainter [14.402778136825642]
  We propose a novel First Frame Filling Video Diffusion Inpainting model (FFF-VDI) We propagate the noise latent information of future frames to fill the masked areas of the first frame's noise latent code. Next, we fine-tune the pre-trained image-to-video diffusion model to generate the inpainted video.
  arXiv  Detail & Related papers  (2024-08-21T08:01:00Z)
• Raformer: Redundancy-Aware Transformer for Video Wire Inpainting [77.41727407673066]
  Video Wire Inpainting (VWI) is a prominent application in video inpainting, aimed at flawlessly removing wires in films or TV series. Wire removal poses greater challenges due to the wires being longer and slimmer than objects typically targeted in general video inpainting tasks. We introduce a new VWI dataset with a novel mask generation strategy, namely Wire Removal Video dataset 2 (WRV2) and Pseudo Wire-Shaped (PWS) Masks. WRV2 dataset comprises over 4,000 videos with an average length of 80 frames, designed to facilitate the development and efficacy of inpainting models.
  arXiv  Detail & Related papers  (2024-04-24T11:02:13Z)
• HumMUSS: Human Motion Understanding using State Space Models [6.821961232645209]
  We propose a novel attention-free model for human motion understanding building upon recent advancements in state space models. Our model supports both offline and real-time applications. For real-time sequential prediction, our model is both memory efficient and several times faster than transformer-based approaches.
  arXiv  Detail & Related papers  (2024-04-16T19:59:21Z)
• Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis [69.83405335645305]
  We argue that naively bringing advances of image models to the video generation domain reduces motion fidelity, visual quality and impairs scalability. In this work, we build Snap Video, a video-first model that systematically addresses these challenges. We show that a U-Net - a workhorse behind image generation - scales poorly when generating videos, requiring significant computational overhead. This allows us to efficiently train a text-to-video model with billions of parameters for the first time, reach state-of-the-art results on a number of benchmarks, and generate videos with substantially higher quality, temporal consistency, and motion complexity.
  arXiv  Detail & Related papers  (2024-02-22T18:55:08Z)
• AVID: Any-Length Video Inpainting with Diffusion Model [30.860927136236374]
  We introduce Any-Length Video Inpainting with Diffusion Model, dubbed as AVID. Our model is equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting. Our experiments show our model can robustly deal with various inpainting types at different video duration ranges, with high quality.
  arXiv  Detail & Related papers  (2023-12-06T18:56:14Z)
• Learning Prior Feature and Attention Enhanced Image Inpainting [63.21231753407192]
  This paper incorporates the pre-training based Masked AutoEncoder (MAE) into the inpainting model. We propose to use attention priors from MAE to make the inpainting model learn more long-distance dependencies between masked and unmasked regions.
  arXiv  Detail & Related papers  (2022-08-03T04:32:53Z)
• Learning Trajectory-Aware Transformer for Video Super-Resolution [50.49396123016185]
  Video super-resolution aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Existing approaches usually align and aggregate video frames from limited adjacent frames. We propose a novel Transformer for Video Super-Resolution (TTVSR)
  arXiv  Detail & Related papers  (2022-04-08T03:37:39Z)
• Learning Joint Spatial-Temporal Transformations for Video Inpainting [58.939131620135235]
  We propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. We simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss.
  arXiv  Detail & Related papers  (2020-07-20T16:35:48Z)
• Latent Video Transformer [30.0340468756089]
  Some generative models for videos require up to 512 Processing Units for parallel training. In this work, we address this problem via modeling the dynamics in a latent space. We demonstrate the performance of our approach on BAIR Robot Pushing Kinetics-600 datasets.
  arXiv  Detail & Related papers  (2020-06-18T17:38:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.