Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latency

• URL: http://arxiv.org/abs/2506.02908v1
• Date: Tue, 03 Jun 2025 14:14:28 GMT
• Title: Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latency
• Authors: Bunlong Lay, Rostilav Makarov, Timo Gerkmann,
• Abstract summary: We adapt a sliding window diffusion framework to the speech enhancement task.<n>Our approach corrupts speech signals through time, assigning more noise to frames close to the present in a buffer.<n>This marks the first practical diffusion-based solution for online speech enhancement.
• Score: 14.001679439460359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models are a class of generative models that have been recently used for speech enhancement with remarkable success but are computationally expensive at inference time. Therefore, these models are impractical for processing streaming data in real-time. In this work, we adapt a sliding window diffusion framework to the speech enhancement task. Our approach progressively corrupts speech signals through time, assigning more noise to frames close to the present in a buffer. This approach outputs denoised frames with a delay proportional to the chosen buffer size, enabling a trade-off between performance and latency. Empirical results demonstrate that our method outperforms standard diffusion models and runs efficiently on a GPU, achieving an input-output latency in the order of 0.3 to 1 seconds. This marks the first practical diffusion-based solution for online speech enhancement.

Related papers

• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion [70.4360995984905]
  We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models.<n>It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs.
  arXiv  Detail & Related papers  (2025-06-09T17:59:55Z)
• Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion [16.99620863197586]
  Diffusion language models offer parallel token generation and inherent bidirectionality.<n>State-of-the-art diffusion models (e.g., Dream 7B, LLaDA 8B) suffer from slow inference.<n>We introduce Guided Diffusion, a training-free method that uses a lightweight pretrained autoregressive model to supervise token unmasking.<n>For the first time, diffusion language models achieve a comparable and even faster latency as the widely adopted autoregressive models.
  arXiv  Detail & Related papers  (2025-05-27T17:39:39Z)
• One-Step Diffusion Model for Image Motion-Deblurring [85.76149042561507]
  We propose a one-step diffusion model for deblurring (OSDD), a novel framework that reduces the denoising process to a single step.<n>To tackle fidelity loss in diffusion models, we introduce an enhanced variational autoencoder (eVAE), which improves structural restoration.<n>Our method achieves strong performance on both full and no-reference metrics.
  arXiv  Detail & Related papers  (2025-03-09T09:39:57Z)
• Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models [64.2445487645478]
  Large Language Models have shown remarkable efficacy in generating streaming data such as text and audio. We present Live2Diff, the first attempt at designing a video diffusion model with uni-directional temporal attention, specifically targeting live streaming video translation.
  arXiv  Detail & Related papers  (2024-07-11T17:34:51Z)
• StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation [29.30999290150683]
  We introduce StreamDiffusion, a real-time diffusion pipeline for interactive image generation. Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction. We present a novel approach that transforms the original sequential denoising into the denoising process.
  arXiv  Detail & Related papers  (2023-12-19T18:18:33Z)
• Single and Few-step Diffusion for Generative Speech Enhancement [18.487296462927034]
  Diffusion models have shown promising results in speech enhancement. In this paper, we address these limitations through a two-stage training approach. We show that our proposed method keeps a steady performance and therefore largely outperforms the diffusion baseline in this setting.
  arXiv  Detail & Related papers  (2023-09-18T11:30:58Z)
• DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection [38.12212015133935]
  A novel framework, DiffusionVMR, is proposed to redefine the two tasks as a unified conditional denoising generation process. Experiments conducted on five widely-used benchmarks demonstrate the effectiveness and flexibility of the proposed DiffusionVMR.
  arXiv  Detail & Related papers  (2023-08-29T08:20:23Z)
• Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion [85.54515118077825]
  This paper proposes a linear diffusion model (LinDiff) based on an ordinary differential equation to simultaneously reach fast inference and high sample quality. To reduce computational complexity, LinDiff employs a patch-based processing approach that partitions the input signal into small patches. Our model can synthesize speech of a quality comparable to that of autoregressive models with faster synthesis speed.
  arXiv  Detail & Related papers  (2023-06-09T07:02:43Z)
• An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initialization [58.88327181933151]
  In this paper, we propose an efficient query-based membership inference attack (MIA) Experimental results indicate that the proposed method can achieve competitive performance with only two queries on both discrete-time and continuous-time diffusion models. To the best of our knowledge, this work is the first to study the robustness of diffusion models to MIA in the text-to-speech task.
  arXiv  Detail & Related papers  (2023-05-26T16:38:48Z)
• Real-time Streaming Video Denoising with Bidirectional Buffers [48.57108807146537]
  Real-time denoising algorithms are typically adopted on the user device to remove the noise involved during the shooting and transmission of video streams. Recent multi-output inference works propagate the bidirectional temporal feature with a parallel or recurrent framework. We propose a Bidirectional Streaming Video Denoising framework, to achieve high-fidelity real-time denoising for streaming videos with both past and future temporal receptive fields.
  arXiv  Detail & Related papers  (2022-07-14T14:01:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.