Deep Video Prior for Video Consistency and Propagation
- URL: http://arxiv.org/abs/2201.11632v1
- Date: Thu, 27 Jan 2022 16:38:52 GMT
- Title: Deep Video Prior for Video Consistency and Propagation
- Authors: Chenyang Lei, Yazhou Xing, Hao Ouyang, Qifeng Chen
- Abstract summary: We present a novel and general approach for blind video temporal consistency.
Our method is only trained on a pair of original and processed videos directly instead of a large dataset.
We show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior.
- Score: 58.250209011891904
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Applying an image processing algorithm independently to each video frame
often leads to temporal inconsistency in the resulting video. To address this
issue, we present a novel and general approach for blind video temporal
consistency. Our method is only trained on a pair of original and processed
videos directly instead of a large dataset. Unlike most previous methods that
enforce temporal consistency with optical flow, we show that temporal
consistency can be achieved by training a convolutional neural network on a
video with Deep Video Prior (DVP). Moreover, a carefully designed iteratively
reweighted training strategy is proposed to address the challenging multimodal
inconsistency problem. We demonstrate the effectiveness of our approach on 7
computer vision tasks on videos. Extensive quantitative and perceptual
experiments show that our approach obtains superior performance than
state-of-the-art methods on blind video temporal consistency. We further extend
DVP to video propagation and demonstrate its effectiveness in propagating three
different types of information (color, artistic style, and object
segmentation). A progressive propagation strategy with pseudo labels is also
proposed to enhance DVP's performance on video propagation. Our source codes
are publicly available at https://github.com/ChenyangLEI/deep-video-prior.
Related papers
- Learning Temporally Consistent Video Depth from Video Diffusion Priors [57.929828486615605]
This work addresses the challenge of video depth estimation.
We reformulate the prediction task into a conditional generation problem.
This allows us to leverage the prior knowledge embedded in existing video generation models.
arXiv Detail & Related papers (2024-06-03T16:20:24Z) - Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
We focus on improving the temporal consistency without introducing overhead in inference.
We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
arXiv Detail & Related papers (2022-02-24T23:51:36Z) - Learning Long-Term Style-Preserving Blind Video Temporal Consistency [6.6908747077585105]
We propose a postprocessing model, to the transformation applied to videos, in the form of a recurrent neural network.
Our model is trained using a Ping Pong procedure and its corresponding loss, recently introduced for GAN video generation.
We evaluate our model on the DAVIS and videvo.net datasets and show that our approach offers state-of-the-art results concerning flicker removal.
arXiv Detail & Related papers (2021-03-12T13:54:34Z) - Semi-Supervised Action Recognition with Temporal Contrastive Learning [50.08957096801457]
We learn a two-pathway temporal contrastive model using unlabeled videos at two different speeds.
We considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods.
arXiv Detail & Related papers (2021-02-04T17:28:35Z) - Blind Video Temporal Consistency via Deep Video Prior [61.062900556483164]
We present a novel and general approach for blind video temporal consistency.
Our method is only trained on a pair of original and processed videos directly.
We show that temporal consistency can be achieved by training a convolutional network on a video with the Deep Video Prior.
arXiv Detail & Related papers (2020-10-22T16:19:20Z) - Self-supervised Video Representation Learning by Pace Prediction [48.029602040786685]
This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction.
It stems from the observation that human visual system is sensitive to video pace.
We randomly sample training clips in different paces and ask a neural network to identify the pace for each video clip.
arXiv Detail & Related papers (2020-08-13T12:40:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.