Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames

• URL: http://arxiv.org/abs/2003.08635v1
• Date: Thu, 19 Mar 2020 09:06:06 GMT
• Title: Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames
• Authors: Osamu Shouno
• Abstract summary: We propose a deep residual network with the hierarchical architecture where each layer makes a prediction of future state at different spatial resolution. We trained our model with adversarial and perceptual loss functions, and evaluated it on a natural video dataset captured by car-mounted cameras.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in deep learning have significantly improved performance of video prediction. However, state-of-the-art methods still suffer from blurriness and distortions in their future predictions, especially when there are large motions between frames. To address these issues, we propose a deep residual network with the hierarchical architecture where each layer makes a prediction of future state at different spatial resolution, and these predictions of different layers are merged via top-down connections to generate future frames. We trained our model with adversarial and perceptual loss functions, and evaluated it on a natural video dataset captured by car-mounted cameras. Our model quantitatively outperforms state-of-the-art baselines in future frame prediction on video sequences of both largely and slightly changing frames. Furthermore, our model generates future frames with finer details and textures that are perceptually more realistic than the baselines, especially under fast camera motions.

Related papers

• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)
• STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction [20.701792842768747]
  We propose a novel video prediction model, which has infinite-dimensional latent variables over the temporal domain. Our model is able to achieve temporal continuous prediction, i.e., predicting in an unsupervised way, with an arbitrarily high frame rate.
  arXiv  Detail & Related papers  (2023-12-11T16:12:43Z)
• HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator [90.74663948713615]
  We train an autoregressive latent video prediction model capable of predicting high-fidelity future frames. We produce high-resolution (256x256) videos with minimal modification to existing models.
  arXiv  Detail & Related papers  (2022-09-15T08:41:57Z)
• Optimizing Video Prediction via Video Frame Interpolation [53.16726447796844]
  We present a new optimization framework for video prediction via video frame, inspired by photo-realistic results of video framescapes. Our framework is based on optimization with a pretrained differentiable video frame module without the need for a training dataset. Our approach outperforms other video prediction methods that require a large amount of training data or extra semantic information.
  arXiv  Detail & Related papers  (2022-06-27T17:03:46Z)
• Video Prediction at Multiple Scales with Hierarchical Recurrent Networks [24.536256844130996]
  We propose a novel video prediction model able to forecast future possible outcomes of different levels of granularity simultaneously. By combining spatial and temporal downsampling, MSPred is able to efficiently predict abstract representations over long time horizons. In our experiments, we demonstrate that our proposed model accurately predicts future video frames as well as other representations on various scenarios.
  arXiv  Detail & Related papers  (2022-03-17T13:08:28Z)
• Local Frequency Domain Transformer Networks for Video Prediction [24.126513851779936]
  Video prediction is of interest not only in anticipating visual changes in the real world but has, above all, emerged as an unsupervised learning rule. This paper proposes a fully differentiable building block that can perform all of those tasks separately while maintaining interpretability.
  arXiv  Detail & Related papers  (2021-05-10T19:48:42Z)
• Learning Semantic-Aware Dynamics for Video Prediction [68.04359321855702]
  We propose an architecture and training scheme to predict video frames by explicitly modeling dis-occlusions. The appearance of the scene is warped from past frames using the predicted motion in co-visible regions.
  arXiv  Detail & Related papers  (2021-04-20T05:00:24Z)
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction [55.4498466252522]
  We set a new standard of video prediction with orders of magnitude longer prediction time than existing approaches. Our method predicts future frames by first estimating a sequence of semantic structures and subsequently translating the structures to pixels by video-to-video translation. We evaluate our method on three challenging datasets involving car driving and human dancing, and demonstrate that it can generate complicated scene structures and motions over a very long time horizon.
  arXiv  Detail & Related papers  (2021-04-14T08:39:38Z)
• Future Video Synthesis with Object Motion Prediction [54.31508711871764]
  Instead of synthesizing images directly, our approach is designed to understand the complex scene dynamics. The appearance of the scene components in the future is predicted by non-rigid deformation of the background and affine transformation of moving objects. Experimental results on the Cityscapes and KITTI datasets show that our model outperforms the state-of-the-art in terms of visual quality and accuracy.
  arXiv  Detail & Related papers  (2020-04-01T16:09:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.