SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation

• URL: http://arxiv.org/abs/2505.19151v1
• Date: Sun, 25 May 2025 13:58:52 GMT
• Title: SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation
• Authors: Shenggan Cheng, Yuanxin Wei, Lansong Diao, Yong Liu, Bujiao Chen, Lianghua Huang, Yu Liu, Wenyuan Yu, Jiangsu Du, Wei Lin, Yang You,
• Abstract summary: SRDiffusion is a novel framework that leverages collaboration between large and small models to reduce inference cost.<n>Our method is introduced as a new direction to existing acceleration strategies, offering a practical solution for scalable video generation.
• Score: 26.045123066151838
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Leveraging the diffusion transformer (DiT) architecture, models like Sora, CogVideoX and Wan have achieved remarkable progress in text-to-video, image-to-video, and video editing tasks. Despite these advances, diffusion-based video generation remains computationally intensive, especially for high-resolution, long-duration videos. Prior work accelerates its inference by skipping computation, usually at the cost of severe quality degradation. In this paper, we propose SRDiffusion, a novel framework that leverages collaboration between large and small models to reduce inference cost. The large model handles high-noise steps to ensure semantic and motion fidelity (Sketching), while the smaller model refines visual details in low-noise steps (Rendering). Experimental results demonstrate that our method outperforms existing approaches, over 3$\times$ speedup for Wan with nearly no quality loss for VBench, and 2$\times$ speedup for CogVideoX. Our method is introduced as a new direction orthogonal to existing acceleration strategies, offering a practical solution for scalable video generation.

Related papers

• Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Caching [57.7533917467934]
  EasyCache is a training-free acceleration framework for video diffusion models.<n>We conduct comprehensive studies on various large-scale video generation models, including OpenSora, Wan2.1, and HunyuanVideo.<n>Our method achieves leading acceleration performance, reducing inference time by up to 2.1-3.3$times$ compared to the original baselines.
  arXiv  Detail & Related papers  (2025-07-03T17:59:54Z)
• FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution [68.77813885751308]
  State-of-the-art (SOTA) compressed video super-resolution (CVSR) models face persistent challenges, including prolonged inference time, complex training pipelines, and reliance on auxiliary information.<n>We propose an efficient and scalable solution inspired by the structural and statistical similarities between hyperspectral images (HSI) and video data.<n>Our approach introduces a compression-driven dimensionality reduction strategy that reduces computational complexity, accelerates inference, and enhances the extraction of temporal information across frames.
  arXiv  Detail & Related papers  (2025-06-13T07:59:52Z)
• DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution [43.83739935393097]
  We propose DOVE, an efficient one-step diffusion model for real-world video super-resolution.<n>DOVE is obtained by fine-tuning a pretrained video diffusion model (*i.e.*, CogVideoX)<n>Experiments show that DOVE exhibits comparable or superior performance to multi-step diffusion-based VSR methods.
  arXiv  Detail & Related papers  (2025-05-22T05:16:45Z)
• VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models [58.464465016269614]
  We propose a novel framework for solving high-definition video inverse problems using latent image diffusion models.<n>Our approach delivers HD-resolution reconstructions in under 6 seconds per frame on a single NVIDIA 4090 GPU.
  arXiv  Detail & Related papers  (2024-11-29T08:10:49Z)
• Fast and Memory-Efficient Video Diffusion Using Streamlined Inference [41.505829393818274]
  Current video diffusion models exhibit demanding computational requirements and high peak memory usage. We present Streamlined Inference, which leverages the temporal and spatial properties of video diffusion models. Our approach significantly reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU.
  arXiv  Detail & Related papers  (2024-11-02T07:52:18Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• SF-V: Single Forward Video Generation Model [57.292575082410785]
  We propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained models. Experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead.
  arXiv  Detail & Related papers  (2024-06-06T17:58:27Z)
• Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting [27.302681897961588]
  Deep convolutional neural networks (DNNs) are widely used in various fields of computer vision. We propose a novel method for high-quality and efficient video resolution upscaling tasks. We deploy our models on an off-the-shelf mobile phone, and experimental results show that our method achieves real-time video super-resolution with high video quality.
  arXiv  Detail & Related papers  (2023-03-15T02:40:02Z)
• Latent Video Diffusion Models for High-Fidelity Long Video Generation [58.346702410885236]
  We introduce lightweight video diffusion models using a low-dimensional 3D latent space. We also propose hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. Our framework generates more realistic and longer videos than previous strong baselines.
  arXiv  Detail & Related papers  (2022-11-23T18:58:39Z)
• Enhanced Quadratic Video Interpolation [56.54662568085176]
  We propose an enhanced quadratic video (EQVI) model to handle more complicated scenes and motion patterns. To further boost the performance, we devise a novel multi-scale fusion network (MS-Fusion) which can be regarded as a learnable augmentation process. The proposed EQVI model won the first place in the AIM 2020 Video Temporal Super-Resolution Challenge.
  arXiv  Detail & Related papers  (2020-09-10T02:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.