Autoregression-free video prediction using diffusion model for mitigating error propagation

• URL: http://arxiv.org/abs/2505.22111v2
• Date: Fri, 30 May 2025 16:09:29 GMT
• Title: Autoregression-free video prediction using diffusion model for mitigating error propagation
• Authors: Woonho Ko, Jin Bok Park, Il Yong Chun,
• Abstract summary: This paper proposes the first AutoRegression-Free (ARFree) video prediction framework using diffusion models.<n>Different from an autoregressive video prediction mechanism, ARFree directly predicts any future frames from the context frame.<n>Our experiments with two benchmark datasets show that the proposed ARFree video prediction framework outperforms several state-of-the-art video prediction methods.
• Score: 4.813333335683417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing long-term video prediction methods often rely on an autoregressive video prediction mechanism. However, this approach suffers from error propagation, particularly in distant future frames. To address this limitation, this paper proposes the first AutoRegression-Free (ARFree) video prediction framework using diffusion models. Different from an autoregressive video prediction mechanism, ARFree directly predicts any future frame tuples from the context frame tuple. The proposed ARFree consists of two key components: 1) a motion prediction module that predicts a future motion using motion feature extracted from the context frame tuple; 2) a training method that improves motion continuity and contextual consistency between adjacent future frame tuples. Our experiments with two benchmark datasets show that the proposed ARFree video prediction framework outperforms several state-of-the-art video prediction methods.

Related papers

• State-space Decomposition Model for Video Prediction Considering Long-term Motion Trend [3.910356300831074]
  We propose a state-space decomposition video prediction model that decomposes the overall video frame generation into deterministic appearance prediction and motion prediction. We infer the long-term motion trend from conditional frames to guide the generation of future frames that exhibit high consistency with the conditional frames.
  arXiv  Detail & Related papers  (2024-04-17T17:19:48Z)
• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)
• STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction [20.701792842768747]
  We propose a novel video prediction model, which has infinite-dimensional latent variables over the temporal domain. Our model is able to achieve temporal continuous prediction, i.e., predicting in an unsupervised way, with an arbitrarily high frame rate.
  arXiv  Detail & Related papers  (2023-12-11T16:12:43Z)
• Motion and Context-Aware Audio-Visual Conditioned Video Prediction [58.9467115916639]
  We decouple the audio-visual conditioned video prediction into motion and appearance modeling. The multimodal motion estimation predicts future optical flow based on the audio-motion correlation. We propose context-aware refinement to address the diminishing of the global appearance context.
  arXiv  Detail & Related papers  (2022-12-09T05:57:46Z)
• A unified model for continuous conditional video prediction [14.685237010856953]
  Conditional video prediction tasks are normally solved by task-related models. Almost all conditional video prediction models can only achieve discrete prediction. In this paper, we propose a unified model that addresses these two issues at the same time.
  arXiv  Detail & Related papers  (2022-10-11T22:26:59Z)
• HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator [90.74663948713615]
  We train an autoregressive latent video prediction model capable of predicting high-fidelity future frames. We produce high-resolution (256x256) videos with minimal modification to existing models.
  arXiv  Detail & Related papers  (2022-09-15T08:41:57Z)
• Optimizing Video Prediction via Video Frame Interpolation [53.16726447796844]
  We present a new optimization framework for video prediction via video frame, inspired by photo-realistic results of video framescapes. Our framework is based on optimization with a pretrained differentiable video frame module without the need for a training dataset. Our approach outperforms other video prediction methods that require a large amount of training data or extra semantic information.
  arXiv  Detail & Related papers  (2022-06-27T17:03:46Z)
• VPTR: Efficient Transformers for Video Prediction [14.685237010856953]
  We propose a new Transformer block for video future frames prediction based on an efficient local spatial-temporal separation attention mechanism. Based on this new Transformer block, a fully autoregressive video future frames prediction Transformer is proposed. A non-autoregressive video prediction Transformer is also proposed to increase the inference speed and reduce the accumulated inference errors of its autoregressive counterpart.
  arXiv  Detail & Related papers  (2022-03-29T18:09:09Z)
• Fourier-based Video Prediction through Relational Object Motion [28.502280038100167]
  Deep recurrent architectures have been applied to the task of video prediction. Here, we explore a different approach by using frequency-domain approaches for video prediction. The resulting predictions are consistent with the observed dynamics in a scene and do not suffer from blur.
  arXiv  Detail & Related papers  (2021-10-12T10:43:05Z)
• Novel View Video Prediction Using a Dual Representation [51.58657840049716]
  Given a set of input video clips from a single/multiple views, our network is able to predict the video from a novel view. The proposed approach does not require any priors and is able to predict the video from wider angular distances, upto 45 degree. A comparison with the State-of-the-art novel view video prediction methods shows an improvement of 26.1% in SSIM, 13.6% in PSNR, and 60% inFVD scores without using explicit priors from target views.
  arXiv  Detail & Related papers  (2021-06-07T20:41:33Z)
• Less is More: Sparse Sampling for Dense Reaction Predictions [60.005266111509435]
  We present our method for 2021 Evoked Expression from Videos Challenge. Our model utilizes both audio and image modalities as inputs to predict emotion changes of viewers. The proposed method has achieved pearson's correlation score of 0.04430 on the final private test set.
  arXiv  Detail & Related papers  (2021-06-03T11:33:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.