Training-free Latent Inter-Frame Pruning with Attention Recovery

• URL: http://arxiv.org/abs/2603.05811v1
• Date: Fri, 06 Mar 2026 01:49:47 GMT
• Title: Training-free Latent Inter-Frame Pruning with Attention Recovery
• Abstract summary: Current video generation models suffer from high computational latency, making real-time applications prohibitively costly.<n>We propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches.<n>Our method increases video editing throughput by $1.45times$, on average achieving 12.2 FPS on an NVIDIA A6000.
• Score: 50.889009147480856
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current video generation models suffer from high computational latency, making real-time applications prohibitively costly. In this paper, we address this limitation by exploiting the temporal redundancy inherent in video latent patches. To this end, we propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches. Additionally, we introduce a novel Attention Recovery mechanism that approximates the attention values of pruned tokens, thereby removing visual artifacts arising from naively applying the pruning method. Empirically, our method increases video editing throughput by $1.45\times$, on average achieving 12.2 FPS on an NVIDIA A6000 compared to the baseline 8.4 FPS. The proposed method does not compromise generation quality and can be seamlessly integrated with the model without additional training. Our approach effectively bridges the gap between traditional compression algorithms and modern generative pipelines.

Related papers

• Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation [2.639628765667673]
  Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications.<n>We observe that video latents encoded via an autoencoder contain redundancy along the temporal axis.<n>We propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput.
  arXiv  Detail & Related papers  (2026-04-26T20:03:11Z)
• Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation [16.611062315689306]
  We propose Hybrid Forcing, which jointly optimize temporal information retention and computational efficiency.<n>Our model achieves real-time, unbounded 832x480 video generation at 29.5 FPS on a single NVIDIA H100 GPU without quantization or model compression.
  arXiv  Detail & Related papers  (2026-04-11T08:54:07Z)
• Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models [58.3184497327891]
  Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences.<n>We present Astrolabe, an efficient online reinforcement learning framework tailored for distilled AR models.
  arXiv  Detail & Related papers  (2026-03-17T18:32:18Z)
• Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention [28.598033369607723]
  textscLight Forcing is a textitfirst sparse attention solution tailored for AR video generation models.<n>It incorporates a textitChunk-Aware Growth mechanism to quantitatively estimate the contribution of each chunk.<n>We also introduce a textit Sparse Attention to capture informative historical and local context in a coarse-to-fine manner.
  arXiv  Detail & Related papers  (2026-02-04T17:41:53Z)
• Bridging Robustness and Efficiency: Real-Time Low-Light Enhancement via Attention U-Net GAN [0.0]
  We propose a hybrid Attention U-Net GAN that provides generative-level texture recovery at edge-deployable speeds.<n>Our method achieves a best-in-class LPIPS score of 0.112 among efficient models.<n>This represents a 40x speedup over latent diffusion models, making our approach suitable for near real-time applications.
  arXiv  Detail & Related papers  (2026-01-10T10:39:22Z)
• Sliding Window Attention for Learned Video Compression [67.57073402826292]
  This work introduces 3D Sliding Window Attention (SWA), a patchless form of local attention.<n>Our method significantly improves rate-distortion performance, achieving Bjorntegaard Delta-rate savings of up to 18.6 %.
  arXiv  Detail & Related papers  (2025-10-04T20:11:43Z)
• PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution [65.09345929328586]
  We propose an innovative approach, called PatchVSR, which integrates a dual-stream adapter for conditional guidance.<n> Experiments demonstrate that our method can synthesize high-fidelity, high-resolution details at the patch level.<n>We can achieve highly competitive 4K VSR based on a 512x512 resolution base model, with extremely high efficiency.
  arXiv  Detail & Related papers  (2025-09-30T09:55:14Z)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion [67.94300151774085]
  We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models.<n>It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs.
  arXiv  Detail & Related papers  (2025-06-09T17:59:55Z)
• Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape [38.76559841681518]
  A huge bottleneck is the attention mechanism where complexity scales quadratically with resolution and video length.<n>Existing techniques fail to preserve visual quality at extremely high sparsity levels and might even incur non-negligible compute overheads.<n>We propose Re-ttention, which implements very high sparse attention for visual generation models.
  arXiv  Detail & Related papers  (2025-05-28T22:39:12Z)
• DeepCache: Accelerating Diffusion Models for Free [65.02607075556742]
  DeepCache is a training-free paradigm that accelerates diffusion models from the perspective of model architecture. DeepCache capitalizes on the inherent temporal redundancy observed in the sequential denoising steps of diffusion models. Under the same throughput, DeepCache effectively achieves comparable or even marginally improved results with DDIM or PLMS.
  arXiv  Detail & Related papers  (2023-12-01T17:01:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.