Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation

• URL: http://arxiv.org/abs/2604.21221v1
• Date: Thu, 23 Apr 2026 02:22:25 GMT
• Title: Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
• Abstract summary: We introduce Sparse Forcing, a training-and-inference paradigm for autoregressive video diffusion models.<n>Sparse Forcing improves long-horizon generation quality while reducing decoding latency.
• Score: 12.765068538856013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Sparse Forcing, a training-and-inference paradigm for autoregressive video diffusion models that improves long-horizon generation quality while reducing decoding latency. Sparse Forcing is motivated by an empirical observation in autoregressive diffusion rollouts: attention concentrates on a persistent subset of salient visual blocks, forming an implicit spatiotemporal memory in the KV cache, and exhibits a locally structured block-sparse pattern within sliding windows. Building on this observation, we propose a trainable native sparsity mechanism that learns to compress, preserve, and update these persistent blocks while restricting computation within each local window to a dynamically selected local neighborhood. To make the approach practical at scale for both training and inference, we further propose Persistent Block-Sparse Attention (PBSA), an efficient GPU kernel that accelerates sparse attention and memory updates for low-latency, memory-efficient decoding. Experiments show that Sparse Forcing improves the VBench score by +0.26 over Self-Forcing on 5-second text-to-video generation while delivering a 1.11-1.17x decoding speedup and 42% lower peak KV-cache footprint. The gains are more pronounced on longer-horizon rollouts, delivering improved visual quality with +0.68 and +2.74 VBench improvements, and 1.22x and 1.27x speedups on 20-second and 1-minute generations, respectively.

Related papers

• PARE: Pruning and Adaptive Routing for Efficient Video Generation [71.54959622788608]
  Video Diffusion Transformers (DiTs) generate high-quality videos but demand substantial compute due to wide blocks, deep architectures, and iterative sampling.<n>Recent methods reduce cost by compressing width, depth, or sampling steps, but typically commit to a fixed architecture that cannot adapt to individual inputs or denoising stages.<n>We propose PARE, which jointly compresses width and depth with structure-aware pruning and input-adaptive routing.
  arXiv  Detail & Related papers  (2026-05-26T17:43:25Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation [16.611062315689306]
  We propose Hybrid Forcing, which jointly optimize temporal information retention and computational efficiency.<n>Our model achieves real-time, unbounded 832x480 video generation at 29.5 FPS on a single NVIDIA H100 GPU without quantization or model compression.
  arXiv  Detail & Related papers  (2026-04-11T08:54:07Z)
• Small Vision-Language Models are Smart Compressors for Long Video Understanding [73.65465038390771]
  Long-form video understanding relies on intent-driven efficiency rather than greedily padded context.<n>We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding.<n>Tests show Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency.
  arXiv  Detail & Related papers  (2026-04-09T11:40:25Z)
• V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models [48.80617385008755]
  Video language models (VideoLLMs) show strong capability in understanding, yet long-context inference is still dominated by massive redundant visual tokens in the prefill stage.<n>We propose V-CAST, a training-free, plug-and-play pruning policy for long-context video inference.
  arXiv  Detail & Related papers  (2026-03-29T11:53:32Z)
• Training-free Latent Inter-Frame Pruning with Attention Recovery [50.889009147480856]
  Current video generation models suffer from high computational latency, making real-time applications prohibitively costly.<n>We propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches.<n>Our method increases video editing throughput by $1.45times$, on average achieving 12.2 FPS on an NVIDIA A6000.
  arXiv  Detail & Related papers  (2026-03-06T01:49:47Z)
• Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention [28.598033369607723]
  textscLight Forcing is a textitfirst sparse attention solution tailored for AR video generation models.<n>It incorporates a textitChunk-Aware Growth mechanism to quantitatively estimate the contribution of each chunk.<n>We also introduce a textit Sparse Attention to capture informative historical and local context in a coarse-to-fine manner.
  arXiv  Detail & Related papers  (2026-02-04T17:41:53Z)
• Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention [37.91838955436801]
  Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines.<n>As generation progresses, the KV cache grows, causing both increasing latency and escalating GPU memory, which in turn restricts usable temporal context and harms long-range consistency.<n>We propose a unified, training-free attention framework for autoregressive diffusion: TempCache compresses the KV cache via temporal correspondence to bound cache growth; AnnCA accelerates cross-attention by selecting frame-relevant prompt tokens using fast approximate nearest neighbor matching; and AnnSA sparsifies self-attention by restricting each query
  arXiv  Detail & Related papers  (2026-02-02T08:31:21Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• BlockVid: Block Diffusion for High-Quality and Consistent Minute-Long Video Generation [44.45173635133032]
  BlockVid is a novel block diffusion framework equipped with semantic-aware sparse KV cache.<n> LV-Bench is a fine-grained benchmark for minute-long videos, complete with new metrics evaluating long-range coherence.
  arXiv  Detail & Related papers  (2025-11-28T08:25:59Z)
• StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation [65.90400162290057]
  Generative models are reshaping the live-streaming industry by redefining how content is created, styled, and delivered.<n>Recent advances in video diffusion have markedly improved temporal consistency and sampling efficiency for offline generation.<n>Live online streaming operates under strict service-level objectives (SLOs): time-to-first-frame must be minimal, and every frame must meet a per-frame deadline with low jitter.
  arXiv  Detail & Related papers  (2025-11-10T18:51:28Z)
• Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generation [50.04968365065964]
  Diffusion-based talking head models generate high-quality, photorealistic videos but suffer from slow inference.<n>We introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP)<n>We also propose Decoupled Foreground Attention (DFA) to further accelerate attention computations.
  arXiv  Detail & Related papers  (2025-08-25T02:58:39Z)
• BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation [27.57431718095974]
  We present BLADE, a data-free joint training framework for video inference.<n>We show remarkable efficiency gains across different scales.<n>On models such as CogVideoX-5B with short video sequence lengths, our framework delivers a robust 8.89x speedup.
  arXiv  Detail & Related papers  (2025-08-14T15:58:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.