Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

• URL: http://arxiv.org/abs/2607.13770v1
• Date: Wed, 15 Jul 2026 12:31:19 GMT
• Title: Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations
• Abstract summary: Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost.<n>Existing acceleration approaches largely inherit attention techniques from sparse models which fail to consider unique video data.<n>This paper presents Kale Kale, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wisetemporal correlations in latent space.<n> Evaluations across three mainstream vDiT models show that Kale Kale achieves 5.9x speedup and 16.0x energy savings over state of the art accelerators.
• Score: 47.69636045010864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.

Related papers

• Towards Error-Free Long Video Generation [56.86952045212838]
  In this paper, we introduce an infinite-length video generation framework that produces high-quality, dynamic, and identity-consistent single-shot long videos.<n>We first finetune a diffusion model as a video extension model on large-scale short video data to autoregressively generate temporally coherent clips.<n>Our framework establishes a new benchmark for realistic and coherent minute-level video synthesis.
  arXiv  Detail & Related papers  (2026-06-21T07:39:37Z)
• PARE: Pruning and Adaptive Routing for Efficient Video Generation [71.54959622788608]
  Video Diffusion Transformers (DiTs) generate high-quality videos but demand substantial compute due to wide blocks, deep architectures, and iterative sampling.<n>Recent methods reduce cost by compressing width, depth, or sampling steps, but typically commit to a fixed architecture that cannot adapt to individual inputs or denoising stages.<n>We propose PARE, which jointly compresses width and depth with structure-aware pruning and input-adaptive routing.
  arXiv  Detail & Related papers  (2026-05-26T17:43:25Z)
• Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep [37.62908191585867]
  HetCache is a training-free diffusion acceleration framework for video-to-video (MV2V) generation and editing.<n>It reduces redundant attention operations while maintaining editing consistency and fidelity.<n> Experiments show that HetCache achieves a noticeable acceleration, including a 2.67$times$ latency speedup and FLOPs reduction over commonly used foundation models.
  arXiv  Detail & Related papers  (2026-03-25T12:53:31Z)
• TIMERIPPLE: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space [15.535854202219072]
  We take a principled approach to accelerate self-attention in vDiTs by leveraging the reuse-temporal correlations in the latent space.<n>We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level.<n>We propose a lightweight and adaptive strategy that approximates attention computations by reusing partial attention scores spatially or temporally correlated tokens along individual channels.
  arXiv  Detail & Related papers  (2025-11-15T05:07:31Z)
• Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformers [29.130090574300635]
  Video diffusion transformers (vDiTs) have made tremendous progress in text-to-video generation, but their compute demands pose a major challenge for practical deployment.<n>We introduce Astraea, a framework that searches for near-optimal configurations for vDiT-based video generation under a performance target.
  arXiv  Detail & Related papers  (2025-06-05T14:41:38Z)
• VORTA: Efficient Video Diffusion via Routing Sparse Attention [54.84294780326206]
  VORTA is an acceleration framework with two novel components.<n>It achieves an end-to-end speedup $1.76times$ without loss of quality on VBench.<n>It can seamlessly integrate with various other acceleration methods, such as model caching and step distillation, reaching up to speedup $14.41times$ with negligible performance degradation.
  arXiv  Detail & Related papers  (2025-05-24T17:46:47Z)
• FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge [60.000984252907195]
  Auto-regressive (AR) models have recently shown promise in visual generation tasks due to their superior sampling efficiency.<n>Video generation requires a substantially larger number of tokens to produce coherent temporal frames, resulting in significant overhead during the decoding phase.<n>We propose the textbfFastCar framework to accelerate the decode phase for the AR video generation by exploring the temporal redundancy.
  arXiv  Detail & Related papers  (2025-05-17T05:00:39Z)
• MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices [24.1144641404561]
  We propose a scheme for exact attention inference acceleration on memory-constrained edge accelerators.<n>We show up to 2.75x speedup and 54% reduction in energy consumption as compared to the state-of-the-art attention fusion method (FLAT) in the edge computing scenario.
  arXiv  Detail & Related papers  (2024-11-20T19:44:26Z)
• SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity [19.900719882624028]
  We propose a memory-efficient scheduling method to eliminate memory overhead and an online adjustment mechanism to minimize accuracy degradation.<n>SparseTem achieves speedup of 1.79x for EfficientDet and 4.72x for CRNN, with minimal accuracy drop and no additional memory overhead.
  arXiv  Detail & Related papers  (2024-10-28T07:13:25Z)
• Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference [95.42299246592756]
  We study the UNet encoder and empirically analyze the encoder features. We find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. We validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation.
  arXiv  Detail & Related papers  (2023-12-15T08:46:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.