DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation

• URL: http://arxiv.org/abs/2606.04432v1
• Date: Wed, 03 Jun 2026 04:25:11 GMT
• Title: DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation
• Abstract summary: Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications.<n>We present DSA, a confidence-guided adaptive framework for AR video diffusion.
• Score: 24.843472730316503
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications. Recent distillation frameworks produce autoregressive video diffusion models with reduced latency, yet these models still use a fixed number of denoising steps per frame, wasting computation on predictable frames and under-refining challenging ones. We present DSA, a confidence-guided adaptive computation framework for AR video diffusion. DSA introduces a lightweight confidence head, trained jointly with the generator under a distribution-matching distillation objective, to estimate per-frame denoising reliability. At inference, this confidence signal dynamically adjusts the number of diffusion steps: simple frames terminate early for speed, while complex frames receive additional refinement. Our method requires no extra video data, no heuristics, and little architectural modification. Experiments show that DSA achieves real-time autoregressive video generation, reaching 22.63 FPS with sub-second latency on H100 GPUs, while maintaining competitive or superior VBench quality compared to recent autoregressive and bidirectional video diffusion models. Our results demonstrate that confidence-guided adaptive sampling provides an effective and practical path toward interactive video generation.

Related papers

• Parallel Decoding Distillation for Fast Image and Video Generation [35.18845060233149]
  We introduce Parallel Decoding Distillation (PDD) for fast inference of diffusion and flow matching models.<n>PDD accelerates generation by predicting multiple denoising steps per network evaluation.<n>Our method achieves SOTA performance with 4-8 NFE on LTX-2.3 Text-to-Video/Audio, Wan 14B Text-to-Video, and Qwen-Image Text-to-Image.
  arXiv  Detail & Related papers  (2026-07-28T17:20:00Z)
• Streaming Autoregressive Video Generation via Diagonal Distillation [50.13573884115673]
  Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity.<n>We propose Diagonal Distillation to exploit temporal information across both video chunks and denoising steps.<n>Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.
  arXiv  Detail & Related papers  (2026-03-10T10:45:24Z)
• VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction [31.191310873846177]
  VideoAR is the first large-scale Visual Autoregressive framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling.<n>VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR with causal next-frame prediction, supported by a 3D multi-scale tokenizer.<n> Empirically, VideoAR achieves new state-of-the-art results improving resolutions among autoregressive models, FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based
  arXiv  Detail & Related papers  (2026-01-09T17:34:59Z)
• Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation [69.57572900337176]
  We introduce Reward Forcing, a novel framework for efficient streaming video generation.<n> EMA-Sink tokens capture both long-term context and recent dynamics, preventing initial frame copying.<n>Re-DMD biases the model's output distribution toward high-reward regions by prioritizing samples with greater dynamics rated by a vision-language model.
  arXiv  Detail & Related papers  (2025-12-04T11:12:13Z)
• POSE: Phased One-Step Adversarial Equilibrium for Video Diffusion Models [18.761042377485367]
  POSE (Phased One-Step Equilibrium) is a distillation framework that reduces the sampling steps of large-scale video diffusion models.<n>We show that POSE outperforms other acceleration methods on VBench-I2V by average 7.15% in semantic alignment, temporal conference and frame quality.
  arXiv  Detail & Related papers  (2025-08-28T17:20:01Z)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion [67.94300151774085]
  We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models.<n>It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs.
  arXiv  Detail & Related papers  (2025-06-09T17:59:55Z)
• Playing with Transformer at 30+ FPS via Next-Frame Diffusion [40.04104312955399]
  Next-Frame Diffusion (NFD) is an autoregressive diffusion transformer that incorporates block-wise causal attention.<n>We show that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency.<n>We achieve autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.
  arXiv  Detail & Related papers  (2025-06-02T07:16:01Z)
• AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset [55.82208863521353]
  We propose AccVideo to reduce the inference steps for accelerating video diffusion models with synthetic dataset.<n>Our model achieves 8.5x improvements in generation speed compared to the teacher model.<n>Compared to previous accelerating methods, our approach is capable of generating videos with higher quality and resolution.
  arXiv  Detail & Related papers  (2025-03-25T08:52:07Z)
• SF-V: Single Forward Video Generation Model [57.292575082410785]
  We propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained models. Experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead.
  arXiv  Detail & Related papers  (2024-06-06T17:58:27Z)
• Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution [65.91317390645163]
  Upscale-A-Video is a text-guided latent diffusion framework for video upscaling. It ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences. It also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation.
  arXiv  Detail & Related papers  (2023-12-11T18:54:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.