Temporal Aware Pruning for Efficient Diffusion-based Video Generation

• URL: http://arxiv.org/abs/2605.17837v2
• Date: Wed, 20 May 2026 21:18:13 GMT
• Title: Temporal Aware Pruning for Efficient Diffusion-based Video Generation
• Abstract summary: We propose TAPE, a training-free Temporal Aware for Efficient diffusion-based video generation.<n>We show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.
• Score: 22.469564305396002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video diffusion models have recently enabled high-quality video generation with ViT-based architectures, but remain computationally intensive because generation requires attention computation over long spatiotemporal sequences. Token pruning has proven effective for ViTs and VLMs. However, most prior pruning methods are attention-based and operate per frame, failing to ensure the vital temporal coherence across frames in video generation tasks. In practice, naively adopting attention-only pruning causes noticeable degradation due to worsened background consistency, flickering, and reduced image quality. To address this, we propose TAPE, a training-free Temporal Aware Pruning for Efficient diffusion-based video generation. TAPE (i) applies temporal smoothing to align token-importance across adjacent frames and suppress selection jitter; and (ii) performs token reselection in selected layers to align token pruning with layers' diverse semantic focus and avoid error accumulation in specific areas; it also (iii) adopt a timestep-level budget scheduling that prunes aggressively at early noisy steps and relaxes pruning during fidelity-critical refinement. The experimental results show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.

Related papers

• Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion [33.237232061796796]
  temporally sparse artifacts such as motion collapse, object flickering, and color oversaturation remain a major barrier to perceptual realism.<n>Existing methods struggle with these issues due to two key limitations.<n>We propose concentrated Implicit Preference Optimization (cIPO), a post-training framework for video diffusion models.
  arXiv  Detail & Related papers  (2026-07-30T11:36:48Z)
• RhymeFlow: Training-Free Acceleration for Video Generation with Asynchronous Denoising Flow Scheduling [51.279397568734424]
  Video generation models based on Diffusion Transformers (DiTs) have achieved remarkable performance in video synthesis.<n>DiTs suffer from high inference latency and computational costs due to the quadratic complexity of 3D attention.<n>We introduce bftextRhymeFlow, a training-free framework that decouples the denoising trajectories of different frames.
  arXiv  Detail & Related papers  (2026-06-04T15:49:37Z)
• ReMATF: Recurrent Motion-Adaptive Multi-scale Turbulence Mitigation for Dynamic Scenes [4.282240730573401]
  Atmospheric turbulence severely degrades video quality by introducing distortions such as geometric warping, blur, and temporal flickering.<n>We propose ReMATF, a lightweight recurrent framework that restores videos using only two frames at a time while preserving spatial detail and temporal stability.
  arXiv  Detail & Related papers  (2026-05-20T17:28:49Z)
• Stream-T1: Test-Time Scaling for Streaming Video Generation [30.575612184373025]
  Stream-T1 is a pioneering comprehensive Test-Time Scaling framework exclusively tailored for streaming video generation.<n> evaluated on both 5s and 30s comprehensive video benchmarks, Stream-T1 demonstrates profound superiority.
  arXiv  Detail & Related papers  (2026-05-06T03:40:05Z)
• Ride the Wave: Precision-Allocated Sparse Attention for Smooth Video Generation [9.132288507881592]
  Video Diffusion Transformers have revolutionized high-fidelity video generation but suffer from the massive computational burden of self-attention.<n>We propose Precision-Allocated Sparse Attention (PASA), a training-free framework designed for highly efficient and temporally smooth video generation.
  arXiv  Detail & Related papers  (2026-04-14T02:51:52Z)
• Streaming Autoregressive Video Generation via Diagonal Distillation [50.13573884115673]
  Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity.<n>We propose Diagonal Distillation to exploit temporal information across both video chunks and denoising steps.<n>Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.
  arXiv  Detail & Related papers  (2026-03-10T10:45:24Z)
• All-in-One Video Restoration under Smoothly Evolving Unknown Weather Degradations [102.94052335735326]
  All-in-one image restoration aims to recover clean images from diverse unknown degradations using a single model.<n>Existing approaches primarily focus on frame-wise degradation variation, overlooking the temporal continuity that naturally exists in real-world degradation processes.<n>We introduce the Smoothly Evolving Unknown Degradations (SEUD) scenario, where both the active degradation set and degradation intensity change continuously over time.
  arXiv  Detail & Related papers  (2026-01-02T02:20:57Z)
• LiteAttention: A Temporal Sparse Attention for Diffusion Transformers [1.3471268811218626]
  LiteAttention exploits temporal coherence to enable evolutionary computation skips across the denoising sequence.<n>We implement a highly optimized LiteAttention kernel on top of FlashAttention and demonstrate substantial speedups on production video diffusion models.
  arXiv  Detail & Related papers  (2025-11-14T08:26:55Z)
• DiTVR: Zero-Shot Diffusion Transformer for Video Restoration [48.97196894658511]
  DiTVR is a zero shot video restoration framework that couples a diffusion transformer with trajectory aware attention and a flow consistent sampler.<n>Our attention mechanism aligns tokens along optical flow trajectories, with particular emphasis on vital layers that exhibit the highest sensitivity to temporal dynamics.<n>The flow guided sampler injects data consistency only into low-frequency bands, preserving high frequency priors while accelerating cache.
  arXiv  Detail & Related papers  (2025-08-11T09:54:45Z)
• Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction [23.493813870675197]
  Event-based video reconstruction has garnered increasing attention due to its advantages, such as high dynamic range and rapid motion capture capabilities. Current methods often prioritize the extraction of temporal information from continuous event flow, leading to an overemphasis on low-frequency texture features in the scene. We introduce a novel approach, the Temporal Residual Guided Diffusion Framework, which effectively leverages both temporal and frequency-based event priors.
  arXiv  Detail & Related papers  (2024-07-15T11:48:57Z)
• Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
  We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
  arXiv  Detail & Related papers  (2021-03-04T12:18:25Z)
• Intrinsic Temporal Regularization for High-resolution Human Video Synthesis [59.54483950973432]
  temporal consistency is crucial for extending image processing pipelines to the video domain. We propose an effective intrinsic temporal regularization scheme, where an intrinsic confidence map is estimated via the frame generator to regulate motion estimation. We apply our intrinsic temporal regulation to single-image generator, leading to a powerful " INTERnet" capable of generating $512times512$ resolution human action videos.
  arXiv  Detail & Related papers  (2020-12-11T05:29:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.