Deep Video Prior for Video Consistency and Propagation

• URL: http://arxiv.org/abs/2201.11632v1
• Date: Thu, 27 Jan 2022 16:38:52 GMT
• Title: Deep Video Prior for Video Consistency and Propagation
• Authors: Chenyang Lei, Yazhou Xing, Hao Ouyang, Qifeng Chen
• Abstract summary: We present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. We show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior.
• Score: 58.250209011891904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Applying an image processing algorithm independently to each video frame often leads to temporal inconsistency in the resulting video. To address this issue, we present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. Unlike most previous methods that enforce temporal consistency with optical flow, we show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior (DVP). Moreover, a carefully designed iteratively reweighted training strategy is proposed to address the challenging multimodal inconsistency problem. We demonstrate the effectiveness of our approach on 7 computer vision tasks on videos. Extensive quantitative and perceptual experiments show that our approach obtains superior performance than state-of-the-art methods on blind video temporal consistency. We further extend DVP to video propagation and demonstrate its effectiveness in propagating three different types of information (color, artistic style, and object segmentation). A progressive propagation strategy with pseudo labels is also proposed to enhance DVP's performance on video propagation. Our source codes are publicly available at https://github.com/ChenyangLEI/deep-video-prior.

Related papers

• Optical-Flow Guided Prompt Optimization for Coherent Video Generation [51.430833518070145]
  We propose a framework called MotionPrompt that guides the video generation process via optical flow. We optimize learnable token embeddings during reverse sampling steps by using gradients from a trained discriminator applied to random frame pairs. This approach allows our method to generate visually coherent video sequences that closely reflect natural motion dynamics, without compromising the fidelity of the generated content.
  arXiv  Detail & Related papers  (2024-11-23T12:26:52Z)
• Learning Temporally Consistent Video Depth from Video Diffusion Priors [57.929828486615605]
  This work addresses the challenge of video depth estimation. We reformulate the prediction task into a conditional generation problem. This allows us to leverage the prior knowledge embedded in existing video generation models.
  arXiv  Detail & Related papers  (2024-06-03T16:20:24Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• Semi-Supervised Action Recognition with Temporal Contrastive Learning [50.08957096801457]
  We learn a two-pathway temporal contrastive model using unlabeled videos at two different speeds. We considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods.
  arXiv  Detail & Related papers  (2021-02-04T17:28:35Z)
• Blind Video Temporal Consistency via Deep Video Prior [61.062900556483164]
  We present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly. We show that temporal consistency can be achieved by training a convolutional network on a video with the Deep Video Prior.
  arXiv  Detail & Related papers  (2020-10-22T16:19:20Z)
• Self-supervised Video Representation Learning by Pace Prediction [48.029602040786685]
  This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace. We randomly sample training clips in different paces and ask a neural network to identify the pace for each video clip.
  arXiv  Detail & Related papers  (2020-08-13T12:40:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.