Speculative Decoding for Autoregressive Video Generation

• URL: http://arxiv.org/abs/2604.17397v1
• Date: Sun, 19 Apr 2026 12:01:57 GMT
• Title: Speculative Decoding for Autoregressive Video Generation
• Abstract summary: We introduce SDVG, which brings speculative decoding to block-based autoregressive video diffusion.<n>A 1.3B drafter proposes candidate blocks via four denoising steps; each block is VAE-decoded and scored by ImageReward.<n>Blocks scoring above a fixed threshold tau are accepted into the 14B target's KV cache; the rest are regenerated by the target.
• Score: 9.370533301840284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive video diffusion is emerging as a promising paradigm for streaming video synthesis, with step distillation serving as the primary means of accelerating inference. Whether speculative decoding, the dominant acceleration strategy for large language models, can be effectively adapted to autoregressive video generation remains an open question, because video blocks are continuous spatiotemporal tensors with no token-level distribution for exact rejection sampling. We introduce SDVG, which brings speculative decoding to block-based autoregressive video diffusion by replacing token verification with an image-quality router. A 1.3B drafter proposes candidate blocks via four denoising steps; each block is VAE-decoded and scored by ImageReward using worst-frame aggregation--taking the minimum per-frame reward to catch single-frame artifacts that averaging would mask. Blocks scoring above a fixed threshold tau are accepted into the 14B target's KV cache; the rest are regenerated by the target. Two additional design choices prove critical: the first block is always force-rejected to anchor scene composition, and tau serves as a single knob that traces a smooth quality-speed Pareto frontier. On 1003 MovieGenVideoBench prompts (832x480), SDVG retains 98.1% of target-only VisionReward quality (0.0773 vs. 0.0788) at a 1.59x speedup with tau=-0.7, and reaches 2.09x at 95.7% quality retention--while consistently outperforming draft-only generation by over +17%. The framework is training-free, requires no architectural changes, and can be seamlessly integrated into existing autoregressive video generation pipelines.

Related papers

• Inference-Time Concept Suppression and Video-Centric Evaluation for Text-to-Video Models [3.414053724037068]
  Text-to-video (T2V) generators can synthesize realistic and temporally coherent videos.<n>Unlike text-to-image concept erasure, T2V unlearning must suppress a target concept that may persist across frames.<n>We propose textbfSIRUS, a training-free inference-time framework for concept-level T2V unlearning.
  arXiv  Detail & Related papers  (2026-07-15T16:26:09Z)
• ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors [52.50325804778549]
  We propose ZeroGVC, a zero-shot generative video compression framework.<n>ZeroGVC encodes the first frame of each group of pictures (GOP) with an image and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression.<n>Experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-lows without any additional training.
  arXiv  Detail & Related papers  (2026-06-21T07:46:14Z)
• Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models [59.36933513963735]
  Diffusion models provide powerful priors for zero-shot video inverse problems.<n>Their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes.<n>We propose Autoregressive Video Inverse problem solver (AVIS) to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-20T02:16:30Z)
• TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos [51.99176811574457]
  Implicit Neural Representations (INRs) have recently demonstrated impressive performance for video compression.<n>However, scaling to high-resolution videos while maintaining encoding efficiency remains a significant challenge.<n>We address these fundamental limitations through three key contributions.<n>We are the first hypernetwork approach to demonstrate results at 480p, 720p and 1080p on UVG, HEVC and MCL-JCV.
  arXiv  Detail & Related papers  (2026-02-18T18:59:55Z)
• Flow caching for autoregressive video generation [72.10021661412364]
  We present FlowCache, the first caching framework specifically designed for autoregressive video generation.<n>Our method achieves remarkable speedups of 2.38 times on MAGI-1 and 6.7 times on SkyReels-V2, with negligible quality degradation.
  arXiv  Detail & Related papers  (2026-02-11T13:11:04Z)
• Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation [69.57572900337176]
  We introduce Reward Forcing, a novel framework for efficient streaming video generation.<n> EMA-Sink tokens capture both long-term context and recent dynamics, preventing initial frame copying.<n>Re-DMD biases the model's output distribution toward high-reward regions by prioritizing samples with greater dynamics rated by a vision-language model.
  arXiv  Detail & Related papers  (2025-12-04T11:12:13Z)
• BlockVid: Block Diffusion for High-Quality and Consistent Minute-Long Video Generation [44.45173635133032]
  BlockVid is a novel block diffusion framework equipped with semantic-aware sparse KV cache.<n> LV-Bench is a fine-grained benchmark for minute-long videos, complete with new metrics evaluating long-range coherence.
  arXiv  Detail & Related papers  (2025-11-28T08:25:59Z)
• BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation [27.57431718095974]
  We present BLADE, a data-free joint training framework for video inference.<n>We show remarkable efficiency gains across different scales.<n>On models such as CogVideoX-5B with short video sequence lengths, our framework delivers a robust 8.89x speedup.
  arXiv  Detail & Related papers  (2025-08-14T15:58:59Z)
• VIDSTAMP: A Temporally-Aware Watermark for Ownership and Integrity in Video Diffusion Models [32.0365189539138]
  VIDSTAMP is a watermarking framework that embeds messages directly into the latent space of temporally-aware video diffusion models.<n>Our method imposes no additional inference cost and offers better perceptual quality than prior methods.
  arXiv  Detail & Related papers  (2025-05-02T17:35:03Z)
• Next Block Prediction: Video Generation via Semi-Autoregressive Modeling [92.60177942930946]
  Next-Block Prediction (NBP) is a semi-autoregressive (semi-AR) framework for video generation.<n>NBP employs bidirectional attention within each block, enabling tokens to capture more robust spatial dependencies.<n>Our model achieves FVD scores of 103.3 on UCF101 and 25.5 on K600, outperforming the vanilla NTP model by an average of 4.4.
  arXiv  Detail & Related papers  (2025-02-11T17:57:53Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [48.35054927704544]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.