Training-free Diffusion Acceleration with Bottleneck Sampling

• URL: http://arxiv.org/abs/2503.18940v2
• Date: Thu, 27 Mar 2025 13:05:19 GMT
• Title: Training-free Diffusion Acceleration with Bottleneck Sampling
• Authors: Ye Tian, Xin Xia, Yuxi Ren, Shanchuan Lin, Xing Wang, Xuefeng Xiao, Yunhai Tong, Ling Yang, Bin Cui,
• Abstract summary: Bottleneck Sampling is a training-free framework that leverages low-resolution priors to reduce computational overhead while preserving output fidelity.<n>It accelerates inference by up to 3$times$ for image generation and 2.5$times$ for video generation, all while maintaining output quality comparable to the standard full-resolution sampling process.
• Score: 37.9135035506567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have demonstrated remarkable capabilities in visual content generation but remain challenging to deploy due to their high computational cost during inference. This computational burden primarily arises from the quadratic complexity of self-attention with respect to image or video resolution. While existing acceleration methods often compromise output quality or necessitate costly retraining, we observe that most diffusion models are pre-trained at lower resolutions, presenting an opportunity to exploit these low-resolution priors for more efficient inference without degrading performance. In this work, we introduce Bottleneck Sampling, a training-free framework that leverages low-resolution priors to reduce computational overhead while preserving output fidelity. Bottleneck Sampling follows a high-low-high denoising workflow: it performs high-resolution denoising in the initial and final stages while operating at lower resolutions in intermediate steps. To mitigate aliasing and blurring artifacts, we further refine the resolution transition points and adaptively shift the denoising timesteps at each stage. We evaluate Bottleneck Sampling on both image and video generation tasks, where extensive experiments demonstrate that it accelerates inference by up to 3$\times$ for image generation and 2.5$\times$ for video generation, all while maintaining output quality comparable to the standard full-resolution sampling process across multiple evaluation metrics.

Related papers

• ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videos [31.597957108232908]
  Video diffusion models (VDMs) facilitate the generation of high-quality videos.<n>Recent studies have uncovered the existence of "golden noises" that can enhance video quality during generation.<n>We propose ScalingNoise, a plug-and-play inference-time search strategy that identifies golden initial noises for the diffusion sampling process.
  arXiv  Detail & Related papers  (2025-03-20T17:54:37Z)
• One-Step Diffusion Model for Image Motion-Deblurring [85.76149042561507]
  We propose a one-step diffusion model for deblurring (OSDD), a novel framework that reduces the denoising process to a single step.<n>To tackle fidelity loss in diffusion models, we introduce an enhanced variational autoencoder (eVAE), which improves structural restoration.<n>Our method achieves strong performance on both full and no-reference metrics.
  arXiv  Detail & Related papers  (2025-03-09T09:39:57Z)
• EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training [15.684865589513597]
  We propose an efficient patch sampling method named EPS for video SR network overfitting. Our method reduces the number of patches for the training to 4% to 25%, depending on the resolution and number of clusters. Compared to the state-of-the-art patch sampling method, EMT, our approach achieves an 83% decrease in overall run time.
  arXiv  Detail & Related papers  (2024-11-25T12:01:57Z)
• Fast constrained sampling in pre-trained diffusion models [77.21486516041391]
  We propose an algorithm that enables fast and high-quality generation under arbitrary constraints. During inference, we can interchange between gradient updates computed on the noisy image and updates computed on the final, clean image. Our approach produces results that rival or surpass the state-of-the-art training-free inference approaches.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy [44.09909260046396]
  We propose AdaptiveDiffusion to reduce noise prediction steps during the denoising process. Our method can significantly speed up the denoising process while generating identical results to the original process, achieving up to an average 25x speedup.
  arXiv  Detail & Related papers  (2024-10-13T15:19:18Z)
• Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference [95.42299246592756]
  We study the UNet encoder and empirically analyze the encoder features. We find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. We validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation.
  arXiv  Detail & Related papers  (2023-12-15T08:46:43Z)
• Compensation Sampling for Improved Convergence in Diffusion Models [12.311434647047427]
  Diffusion models achieve remarkable quality in image generation, but at a cost. Iterative denoising requires many time steps to produce high fidelity images. We argue that the denoising process is crucially limited by an accumulation of the reconstruction error due to an initial inaccurate reconstruction of the target data.
  arXiv  Detail & Related papers  (2023-12-11T10:39:01Z)
• Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner [84.97253871387028]
  A diffusion model, which is formulated to produce an image using thousands of denoising steps, usually suffers from a slow inference speed. We propose a timestep aligner that helps find a more accurate integral direction for a particular interval at the minimum cost. Experiments show that our plug-in design can be trained efficiently and boost the inference performance of various state-of-the-art acceleration methods.
  arXiv  Detail & Related papers  (2023-10-14T02:19:07Z)
• RL-based Stateful Neural Adaptive Sampling and Denoising for Real-Time Path Tracing [1.534667887016089]
  MonteCarlo path tracing is a powerful technique for realistic image synthesis but suffers from high levels of noise at low sample counts. We propose a framework with end-to-end training of a sampling importance network, a latent space encoder network, and a denoiser network.
  arXiv  Detail & Related papers  (2023-10-05T12:39:27Z)
• ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolution [84.73658185158222]
  We propose a diffusion model-based super-resolution method called ACDMSR. Our method adapts the standard diffusion model to perform super-resolution through a deterministic iterative denoising process. Our approach generates more visually realistic counterparts for low-resolution images, emphasizing its effectiveness in practical scenarios.
  arXiv  Detail & Related papers  (2023-07-03T06:49:04Z)
• Simultaneous Image-to-Zero and Zero-to-Noise: Diffusion Models with Analytical Image Attenuation [53.04220377034574]
  We propose incorporating an analytical image attenuation process into the forward diffusion process for high-quality (un)conditioned image generation.<n>Our method represents the forward image-to-noise mapping as simultaneous textitimage-to-zero mapping and textitzero-to-noise mapping.<n>We have conducted experiments on unconditioned image generation, textite.g., CIFAR-10 and CelebA-HQ-256, and image-conditioned downstream tasks such as super-resolution, saliency detection, edge detection, and image inpainting.
  arXiv  Detail & Related papers  (2023-06-23T18:08:00Z)
• Investigating Tradeoffs in Real-World Video Super-Resolution [90.81396836308085]
  Real-world video super-resolution (VSR) models are often trained with diverse degradations to improve generalizability. To alleviate the first tradeoff, we propose a degradation scheme that reduces up to 40% of training time without sacrificing performance. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences.
  arXiv  Detail & Related papers  (2021-11-24T18:58:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.