Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation

• URL: http://arxiv.org/abs/2506.09350v1
• Date: Wed, 11 Jun 2025 03:04:23 GMT
• Title: Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
• Authors: Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, Lu Jiang,
• Abstract summary: Existing large-scale video generation models are computationally intensive.<n>We propose autoregressive adversarial post-training to transform a pre-trained latent video diffusion model into a real-time, interactive video generator.<n>Our model autoregressively generates a latent frame at a time using a single neural function evaluation.
• Score: 35.96336774165454
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing large-scale video generation models are computationally intensive, preventing adoption in real-time and interactive applications. In this work, we propose autoregressive adversarial post-training (AAPT) to transform a pre-trained latent video diffusion model into a real-time, interactive video generator. Our model autoregressively generates a latent frame at a time using a single neural function evaluation (1NFE). The model can stream the result to the user in real time and receive interactive responses as controls to generate the next latent frame. Unlike existing approaches, our method explores adversarial training as an effective paradigm for autoregressive generation. This not only allows us to design an architecture that is more efficient for one-step generation while fully utilizing the KV cache, but also enables training the model in a student-forcing manner that proves to be effective in reducing error accumulation during long video generation. Our experiments demonstrate that our 8B model achieves real-time, 24fps, streaming video generation at 736x416 resolution on a single H100, or 1280x720 on 8xH100 up to a minute long (1440 frames). Visit our research website at https://seaweed-apt.com/2

Related papers

• StreamDiT: Real-Time Streaming Text-to-Video Generation [40.441404889974294]
  This paper proposes StreamDiT, a streaming video generation model.<n>StreamDiT training is based on flow matching by adding a moving buffer.<n>We design mixed training with different partitioning schemes of buffered frames to boost both content consistency and visual quality.<n>Our model enables real-time applications, e.g. streaming generation, interactive generation, and video-to-video.
  arXiv  Detail & Related papers  (2025-07-04T18:00:01Z)
• Training-Free Motion Customization for Distilled Video Generators with Adaptive Test-Time Distillation [53.877572078307935]
  Distilled video generation models offer fast and efficient but struggle with motion customization when guided by reference videos.<n>We propose MotionEcho, a training-free test-time distillation framework that enables motion customization by leveraging diffusion teacher forcing.
  arXiv  Detail & Related papers  (2025-06-24T06:20:15Z)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion [70.4360995984905]
  We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models.<n>It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs.
  arXiv  Detail & Related papers  (2025-06-09T17:59:55Z)
• LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model [22.92353994818742]
  Driving world models are used to simulate futures by video generation based on the condition of the current state and actions.<n>Recent studies utilize the Diffusion Transformer (DiT) as the backbone of driving world models to improve learning flexibility.<n>We propose several solutions to build a simple yet effective long-term driving world model.
  arXiv  Detail & Related papers  (2025-06-02T11:19:23Z)
• Autoregressive Video Generation without Vector Quantization [90.87907377618747]
  We reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction.<n>With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA.<n>Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity.
  arXiv  Detail & Related papers  (2024-12-18T18:59:53Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [52.32078428442281]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• SF-V: Single Forward Video Generation Model [57.292575082410785]
  We propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained models. Experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead.
  arXiv  Detail & Related papers  (2024-06-06T17:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.