StreamDiT: Real-Time Streaming Text-to-Video Generation

• URL: http://arxiv.org/abs/2507.03745v2
• Date: Tue, 08 Jul 2025 03:10:13 GMT
• Title: StreamDiT: Real-Time Streaming Text-to-Video Generation
• Authors: Akio Kodaira, Tingbo Hou, Ji Hou, Masayoshi Tomizuka, Yue Zhao,
• Abstract summary: This paper proposes StreamDiT, a streaming video generation model.<n>StreamDiT training is based on flow matching by adding a moving buffer.<n>We design mixed training with different partitioning schemes of buffered frames to boost both content consistency and visual quality.<n>Our model enables real-time applications, e.g. streaming generation, interactive generation, and video-to-video.
• Score: 40.441404889974294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, great progress has been achieved in text-to-video (T2V) generation by scaling transformer-based diffusion models to billions of parameters, which can generate high-quality videos. However, existing models typically produce only short clips offline, restricting their use cases in interactive and real-time applications. This paper addresses these challenges by proposing StreamDiT, a streaming video generation model. StreamDiT training is based on flow matching by adding a moving buffer. We design mixed training with different partitioning schemes of buffered frames to boost both content consistency and visual quality. StreamDiT modeling is based on adaLN DiT with varying time embedding and window attention. To practice the proposed method, we train a StreamDiT model with 4B parameters. In addition, we propose a multistep distillation method tailored for StreamDiT. Sampling distillation is performed in each segment of a chosen partitioning scheme. After distillation, the total number of function evaluations (NFEs) is reduced to the number of chunks in a buffer. Finally, our distilled model reaches real-time performance at 16 FPS on one GPU, which can generate video streams at 512p resolution. We evaluate our method through both quantitative metrics and human evaluation. Our model enables real-time applications, e.g. streaming generation, interactive generation, and video-to-video. We provide video results and more examples in our project website: https://cumulo-autumn.github.io/StreamDiT/

Related papers

• Playing with Transformer at 30+ FPS via Next-Frame Diffusion [40.04104312955399]
  Next-Frame Diffusion (NFD) is an autoregressive diffusion transformer that incorporates block-wise causal attention.<n>We show that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency.<n>We achieve autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.
  arXiv  Detail & Related papers  (2025-06-02T07:16:01Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [52.32078428442281]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction [43.16308241800144]
  We introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames.<n>We establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101.
  arXiv  Detail & Related papers  (2024-12-06T10:34:50Z)
• Pyramidal Flow Matching for Efficient Video Generative Modeling [67.03504440964564]
  This work introduces a unified pyramidal flow matching algorithm.<n>It sacrifices the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution.<n>The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT)
  arXiv  Detail & Related papers  (2024-10-08T12:10:37Z)
• xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations [120.52120919834988]
  xGen-SynVideo-1 is a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens. DiT model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios.
  arXiv  Detail & Related papers  (2024-08-22T17:55:22Z)
• SF-V: Single Forward Video Generation Model [57.292575082410785]
  We propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained models. Experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead.
  arXiv  Detail & Related papers  (2024-06-06T17:58:27Z)
• Real-time Online Video Detection with Temporal Smoothing Transformers [4.545986838009774]
  A good streaming recognition model captures both long-term dynamics and short-term changes of video. To address this issue, we reformulate the cross-attention in a video transformer through the lens of kernel. We build TeSTra, a Temporal Smoothing Transformer, that takes in arbitrarily long inputs with constant caching and computing overhead.
  arXiv  Detail & Related papers  (2022-09-19T17:59:02Z)
• Representation Recycling for Streaming Video Analysis [19.068248496174903]
  StreamDEQ aims to infer frame-wise representations on videos with minimal per-frame computation. We show that StreamDEQ is able to recover near-optimal representations in a few frames' time and maintain an up-to-date representation throughout the video duration.
  arXiv  Detail & Related papers  (2022-04-28T13:35:14Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• PGT: A Progressive Method for Training Models on Long Videos [45.935259079953255]
  Main-stream method is to split a raw video into clips, leading to incomplete temporal information flow. Inspired by natural language processing techniques dealing with long sentences, we propose to treat videos as serial fragments satisfying Markov property. We empirically demonstrate that it yields significant performance improvements on different models and datasets.
  arXiv  Detail & Related papers  (2021-03-21T06:15:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.