Open-Source Acceleration of Stable-Diffusion.cpp Deployable on All Devices

• URL: http://arxiv.org/abs/2412.05781v3
• Date: Tue, 07 Jan 2025 20:27:09 GMT
• Title: Open-Source Acceleration of Stable-Diffusion.cpp Deployable on All Devices
• Authors: Jingxu Ng, Cheng Lv, Pu Zhao, Wei Niu, Juyi Lin, Minzhou Pan, Yun Liang, Yanzhi Wang,
• Abstract summary: stable-diffusion.Turbo (Sd) emerges as an efficient inference framework to accelerate the diffusion models. In this work, we present an optimized version of Sd leveraging the Winograd algorithm to accelerate 2D convolution operations. We demonstrate a speedup up to 2.76x for individual convolutional layers and an inference speedup up to 4.79x for the overall image generation process.
• Score: 28.774856591172902
• License:
• Abstract: Stable diffusion plays a crucial role in generating high-quality images. However, image generation is time-consuming and memory-intensive. To address this, stable-diffusion.cpp (Sdcpp) emerges as an efficient inference framework to accelerate the diffusion models. Although it is lightweight, the current implementation of ggml_conv_2d operator in Sdcpp is suboptimal, exhibiting both high inference latency and massive memory usage. To address this, in this work, we present an optimized version of Sdcpp leveraging the Winograd algorithm to accelerate 2D convolution operations, which is the primary bottleneck in the pipeline. By analyzing both dependent and independent computation graphs, we exploit the device's locality and parallelism to achieve substantial performance improvements. Our framework delivers correct end-to-end results across various stable diffusion models, including SDv1.4, v1.5, v2.1, SDXL, and SDXL-Turbo. Our evaluation results demonstrate a speedup up to 2.76x for individual convolutional layers and an inference speedup up to 4.79x for the overall image generation process, compared with the original Sdcpp on M1 pro. Homepage: https://github.com/SealAILab/stable-diffusion-cpp

Related papers

• Partially Conditioned Patch Parallelism for Accelerated Diffusion Model Inference [0.7619404259039282]
  Diffusion models have exhibited exciting capabilities in generating images and are also very promising for video creation. The sequential denoising steps required for generating a single sample could take tens or hundreds of iterations. We propose Partially Conditioned Patch Parallelism to accelerate the inference of high-resolution diffusion models.
  arXiv  Detail & Related papers  (2024-12-04T02:12:50Z)
• Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient [52.96232442322824]
  Collaborative Decoding (CoDe) is a novel efficient decoding strategy tailored for the Visual Auto-Regressive ( VAR) framework. CoDe capitalizes on two critical observations: the substantially reduced parameter demands at larger scales and the exclusive generation patterns across different scales. CoDe achieves a 1.7x speedup, slashes memory usage by around 50%, and preserves image quality with only a negligible FID increase from 1.95 to 1.98.
  arXiv  Detail & Related papers  (2024-11-26T15:13:15Z)
• Binarized Diffusion Model for Image Super-Resolution [61.963833405167875]
  Binarization, an ultra-compression algorithm, offers the potential for effectively accelerating advanced diffusion models (DMs) Existing binarization methods result in significant performance degradation. We introduce a novel binarized diffusion model, BI-DiffSR, for image SR.
  arXiv  Detail & Related papers  (2024-06-09T10:30:25Z)
• Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation [6.755999813057451]
  We propose an efficient, fast, and versatile distillation method to accelerate the generation of pre-trained diffusion models: Flash Diffusion. The method reaches state-of-the-art performances in terms of FID and CLIP-Score for few steps image generation on the COCO2014 and COCO 2017 datasets. The versatility of the method is also exposed across several tasks such as text-to-image, inpainting, face-swapping, super-resolution and using different backbones such as UNet-based denoisers (SD1.5, SDXL) or DiT (Pixart-$alpha$
  arXiv  Detail & Related papers  (2024-06-04T14:23:27Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Clockwork Diffusion: Efficient Generation With Model-Step Distillation [42.01130983628078]
  Clockwork Diffusion is a method that periodically reuses computation from preceding denoising steps to approximate low-res feature maps at one or more subsequent steps. For both text-to-image generation and image editing, we demonstrate that Clockwork leads to comparable or improved perceptual scores with drastically reduced computational complexity.
  arXiv  Detail & Related papers  (2023-12-13T13:30:27Z)
• One-step Diffusion with Distribution Matching Distillation [54.723565605974294]
  We introduce Distribution Matching Distillation (DMD), a procedure to transform a diffusion model into a one-step image generator. We enforce the one-step image generator match the diffusion model at distribution level, by minimizing an approximate KL divergence. Our method outperforms all published few-step diffusion approaches, reaching 2.62 FID on ImageNet 64x64 and 11.49 FID on zero-shot COCO-30k.
  arXiv  Detail & Related papers  (2023-11-30T18:59:20Z)
• ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schrödinger Bridge [63.00793292863]
  ToddlerDiffusion is a novel approach to decomposing the complex task of RGB image generation into simpler, interpretable stages. Our method, termed ToddlerDiffusion, cascades modality-specific models, each responsible for generating an intermediate representation. ToddlerDiffusion consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-24T15:20:01Z)
• Fast and High-Quality Image Denoising via Malleable Convolutions [72.18723834537494]
  We present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. Unlike previous works, MalleConv generates a much smaller set of spatially-varying kernels from input. We also build an efficient denoising network using MalleConv, coined as MalleNet.
  arXiv  Detail & Related papers  (2022-01-02T18:35:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.