Progressive Autoregressive Video Diffusion Models

• URL: http://arxiv.org/abs/2410.08151v2
• Date: Sun, 18 May 2025 06:43:40 GMT
• Title: Progressive Autoregressive Video Diffusion Models
• Authors: Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, Yang Zhou,
• Abstract summary: We introduce a more natural formulation of autoregressive long video generation by revisiting the noise level assumption in video diffusion models.<n>Our key idea is to assign the frames with per-frame, progressively increasing noise levels rather than a single noise level and denoise.<n>Video diffusion models equipped with our progressive noise schedule can autoregressively generate long videos with much improved fidelity compared to the baselines.
• Score: 24.97019070991881
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current frontier video diffusion models have demonstrated remarkable results at generating high-quality videos. However, they can only generate short video clips, normally around 10 seconds or 240 frames, due to computation limitations during training. Existing methods naively achieve autoregressive long video generation by directly placing the ending of the previous clip at the front of the attention window as conditioning, which leads to abrupt scene changes, unnatural motion, and error accumulation. In this work, we introduce a more natural formulation of autoregressive long video generation by revisiting the noise level assumption in video diffusion models. Our key idea is to 1. assign the frames with per-frame, progressively increasing noise levels rather than a single noise level and 2. denoise and shift the frames in small intervals rather than all at once. This allows for smoother attention correspondence among frames with adjacent noise levels, larger overlaps between the attention windows, and better propagation of information from the earlier to the later frames. Video diffusion models equipped with our progressive noise schedule can autoregressively generate long videos with much improved fidelity compared to the baselines and minimal quality degradation over time. We present the first results on text-conditioned 60-second (1440 frames) long video generation at a quality close to frontier models. Code and video results are available at https://desaixie.github.io/pa-vdm/.

Related papers

• DiffuseSlide: Training-Free High Frame Rate Video Generation Diffusion [4.863177884263436]
  We present a training-free approach for high FPS video generation using pre-trained diffusion models.<n>Our method, DiffuseSlide, introduces a new pipeline that leverages key frames from low FPS videos and applies innovative techniques, including noise re-injection and sliding window latent denoising.<n>Through extensive experiments, we demonstrate that our approach significantly improves video quality, offering enhanced temporal coherence and spatial fidelity.
  arXiv  Detail & Related papers  (2025-06-02T09:12:41Z)
• Generative Video Bi-flow [14.053608981988793]
  We propose a novel generative video model by robustly learning temporal change as a neural Ordinary Differential Equation (ODE) flow.<n>We demonstrate unconditional video generation in a streaming manner for various video datasets.
  arXiv  Detail & Related papers  (2025-03-09T00:03:59Z)
• Video Diffusion Models are Strong Video Inpainter [14.402778136825642]
  We propose a novel First Frame Filling Video Diffusion Inpainting model (FFF-VDI) We propagate the noise latent information of future frames to fill the masked areas of the first frame's noise latent code. Next, we fine-tune the pre-trained image-to-video diffusion model to generate the inpainted video.
  arXiv  Detail & Related papers  (2024-08-21T08:01:00Z)
• ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models [66.84478240757038]
  A majority of video diffusion models (VDMs) generate long videos in an autoregressive manner, i.e., generating subsequent clips conditioned on last frames of previous clip. We introduce causal (i.e., unidirectional) generation into VDMs, and use past frames as prompt to generate future frames. Our ViD-GPT achieves state-of-the-art performance both quantitatively and qualitatively on long video generation.
  arXiv  Detail & Related papers  (2024-06-16T15:37:22Z)
• SF-V: Single Forward Video Generation Model [57.292575082410785]
  We propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained models. Experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead.
  arXiv  Detail & Related papers  (2024-06-06T17:58:27Z)
• StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text [58.49820807662246]
  We introduce StreamingT2V, an autoregressive approach for long video generation of 80, 240, 600, 1200 or more frames with smooth transitions.<n>Our code will be available at: https://github.com/Picsart-AI-Research/StreamingT2V.
  arXiv  Detail & Related papers  (2024-03-21T18:27:29Z)
• Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition [124.41196697408627]
  We propose content-motion latent diffusion model (CMD), a novel efficient extension of pretrained image diffusion models for video generation. CMD encodes a video as a combination of a content frame (like an image) and a low-dimensional motion latent representation. We generate the content frame by fine-tuning a pretrained image diffusion model, and we generate the motion latent representation by training a new lightweight diffusion model.
  arXiv  Detail & Related papers  (2024-03-21T05:48:48Z)
• SmoothVideo: Smooth Video Synthesis with Noise Constraints on Diffusion Models for One-shot Video Tuning [18.979299814757997]
  One-shot video tuning methods produce videos marred by incoherence and inconsistency. This paper introduces a simple yet effective noise constraint across video frames. By applying the loss to existing one-shot video tuning methods, we significantly improve the overall consistency and smoothness of the generated videos.
  arXiv  Detail & Related papers  (2023-11-29T11:14:43Z)
• FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling [85.60543452539076]
  Existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. This study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. We propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models.
  arXiv  Detail & Related papers  (2023-10-23T17:59:58Z)
• Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models [52.93036326078229]
  Off-the-shelf billion-scale datasets for image generation are available, but collecting similar video data of the same scale is still challenging. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. Our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks.
  arXiv  Detail & Related papers  (2023-05-17T17:59:16Z)
• Latent Video Diffusion Models for High-Fidelity Long Video Generation [58.346702410885236]
  We introduce lightweight video diffusion models using a low-dimensional 3D latent space. We also propose hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. Our framework generates more realistic and longer videos than previous strong baselines.
  arXiv  Detail & Related papers  (2022-11-23T18:58:39Z)
• FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis [77.06890315052563]
  We propose FastLTS, a non-autoregressive end-to-end model which can directly synthesize high-quality speech audios from unconstrained talking videos with low latency. Experiments show that our model achieves $19.76times$ speedup for audio generation compared with the current autoregressive model on input sequences of 3 seconds.
  arXiv  Detail & Related papers  (2022-07-08T10:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.