VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction

• URL: http://arxiv.org/abs/2601.05966v2
• Date: Wed, 14 Jan 2026 14:12:54 GMT
• Title: VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction
• Abstract summary: VideoAR is the first large-scale Visual Autoregressive framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling.<n>VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR with causal next-frame prediction, supported by a 3D multi-scale tokenizer.<n> Empirically, VideoAR achieves new state-of-the-art results improving resolutions among autoregressive models, FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based
• Score: 31.191310873846177
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in video generation have been dominated by diffusion and flow-matching models, which produce high-quality results but remain computationally intensive and difficult to scale. In this work, we introduce VideoAR, the first large-scale Visual Autoregressive (VAR) framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling. VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR modeling with causal next-frame prediction, supported by a 3D multi-scale tokenizer that efficiently encodes spatio-temporal dynamics. To improve long-term consistency, we propose Multi-scale Temporal RoPE, Cross-Frame Error Correction, and Random Frame Mask, which collectively mitigate error propagation and stabilize temporal coherence. Our multi-stage pretraining pipeline progressively aligns spatial and temporal learning across increasing resolutions and durations. Empirically, VideoAR achieves new state-of-the-art results among autoregressive models, improving FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps by over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based models an order of magnitude larger. These results demonstrate that VideoAR narrows the performance gap between autoregressive and diffusion paradigms, offering a scalable, efficient, and temporally consistent foundation for future video generation research.

Related papers

• Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model [33.214927542512555]
  Diff-VF is a framework that converts short-video diffusion backbones into long-video generators without modifying or fine-tuning the base model.<n>VBench-Long evaluation results show that Diff-VF achieves a more favorable balance between temporal coherence and motion diversity than base models.
  arXiv  Detail & Related papers  (2026-08-06T12:53:46Z)
• Mitigating Compounding Error via Video Representation Regularization [32.14962855907007]
  Video diffusion-based world models enable long autoregressive video generation for robotics, autonomous driving and simulation tasks.<n>sliding-window autoregressive inference suffers from severe error accumulation that degrades frame quality over time.<n>We investigate the internal representation dynamics of video world models and discover that compounding error is tightly coupled with dimensional collapse of hidden representations.<n>We propose video representation regularization, a lightweight training constraint that stabilizes latent representations and suppresses iterative error accumulation.
  arXiv  Detail & Related papers  (2026-07-29T15:29:39Z)
• Next Forcing: Causal World Modeling with Multi-Chunk Prediction [65.05727210375643]
  Next Forcing is a multi-chunk prediction framework for causal world modeling.<n>It enables faster training, higher accuracy, and accelerated inference.<n>It achieves 93.1% relative improvement over LingBot-VA at 5k training steps and 2.3x faster convergence.
  arXiv  Detail & Related papers  (2026-06-09T17:59:22Z)
• DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation [24.843472730316503]
  Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications.<n>We present DSA, a confidence-guided adaptive framework for AR video diffusion.
  arXiv  Detail & Related papers  (2026-06-03T04:25:11Z)
• Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers [95.68243351895107]
  We propose a holistic, video-centric paradigm named textbfLocal textbfDiffusion textbfForcing for textbfVideo textbfFrame textbfInterpolation (LDF-VFI)<n>Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence.<n>LDF-VFI achieves state-of-the-art performance on challenging long-sequence benchmarks, demonstrating superior per
  arXiv  Detail & Related papers  (2026-01-21T12:58:52Z)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion [67.94300151774085]
  We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models.<n>It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs.
  arXiv  Detail & Related papers  (2025-06-09T17:59:55Z)
• Playing with Transformer at 30+ FPS via Next-Frame Diffusion [40.04104312955399]
  Next-Frame Diffusion (NFD) is an autoregressive diffusion transformer that incorporates block-wise causal attention.<n>We show that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency.<n>We achieve autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.
  arXiv  Detail & Related papers  (2025-06-02T07:16:01Z)
• Generative Pre-trained Autoregressive Diffusion Transformer [74.25668109048418]
  GPDiT is a Generative Pre-trained Autoregressive Diffusion Transformer.<n>It unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis.<n>It autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics.
  arXiv  Detail & Related papers  (2025-05-12T08:32:39Z)
• AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion [19.98565541640125]
  We introduce Auto-Regressive Diffusion (AR-Diffusion), a novel model that combines the strengths of auto-regressive and diffusion models for flexible video generation.<n>Inspired by auto-regressive generation, we incorporate a non-decreasing constraint on the corruption timesteps of individual frames.<n>This setup, together with temporal causal attention, enables flexible generation of videos with varying lengths while preserving temporal coherence.
  arXiv  Detail & Related papers  (2025-03-10T15:05:59Z)
• Autoregressive Video Generation without Vector Quantization [90.87907377618747]
  We reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction.<n>With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA.<n>Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity.
  arXiv  Detail & Related papers  (2024-12-18T18:59:53Z)
• Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution [65.91317390645163]
  Upscale-A-Video is a text-guided latent diffusion framework for video upscaling. It ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences. It also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation.
  arXiv  Detail & Related papers  (2023-12-11T18:54:52Z)
• Video Probabilistic Diffusion Models in Projected Latent Space [75.4253202574722]
  We propose a novel generative model for videos, coined projected latent video diffusion models (PVDM) PVDM learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources.
  arXiv  Detail & Related papers  (2023-02-15T14:22:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.