Speeding Up Speech Synthesis In Diffusion Models By Reducing Data Distribution Recovery Steps Via Content Transfer

• URL: http://arxiv.org/abs/2309.09652v3
• Date: Sat, 23 Nov 2024 02:05:35 GMT
• Title: Speeding Up Speech Synthesis In Diffusion Models By Reducing Data Distribution Recovery Steps Via Content Transfer
• Authors: Peter Ochieng,
• Abstract summary: Diffusion based vocoders have been criticised for being slow due to the many steps required during sampling. We propose a setup where the targets are the different outputs of forward process time steps. We show through extensive evaluation that the proposed technique generates high-fidelity speech in competitive time.
• Score: 3.2634122554914002
• License:
• Abstract: Diffusion based vocoders have been criticised for being slow due to the many steps required during sampling. Moreover, the model's loss function that is popularly implemented is designed such that the target is the original input $x_0$ or error $\epsilon_0$. For early time steps of the reverse process, this results in large prediction errors, which can lead to speech distortions and increase the learning time. We propose a setup where the targets are the different outputs of forward process time steps with a goal to reduce the magnitude of prediction errors and reduce the training time. We use the different layers of a neural network (NN) to perform denoising by training them to learn to generate representations similar to the noised outputs in the forward process of the diffusion. The NN layers learn to progressively denoise the input in the reverse process until finally the final layer estimates the clean speech. To avoid 1:1 mapping between layers of the neural network and the forward process steps, we define a skip parameter $\tau>1$ such that an NN layer is trained to cumulatively remove the noise injected in the $\tau$ steps in the forward process. This significantly reduces the number of data distribution recovery steps and, consequently, the time to generate speech. We show through extensive evaluation that the proposed technique generates high-fidelity speech in competitive time that outperforms current state-of-the-art tools. The proposed technique is also able to generalize well to unseen speech.

Related papers

• Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner [84.97253871387028]
  A diffusion model, which is formulated to produce an image using thousands of denoising steps, usually suffers from a slow inference speed. We propose a timestep aligner that helps find a more accurate integral direction for a particular interval at the minimum cost. Experiments show that our plug-in design can be trained efficiently and boost the inference performance of various state-of-the-art acceleration methods.
  arXiv  Detail & Related papers  (2023-10-14T02:19:07Z)
• Single and Few-step Diffusion for Generative Speech Enhancement [18.487296462927034]
  Diffusion models have shown promising results in speech enhancement. In this paper, we address these limitations through a two-stage training approach. We show that our proposed method keeps a steady performance and therefore largely outperforms the diffusion baseline in this setting.
  arXiv  Detail & Related papers  (2023-09-18T11:30:58Z)
• UDPM: Upsampling Diffusion Probabilistic Models [33.51145642279836]
  Denoising Diffusion Probabilistic Models (DDPM) have recently gained significant attention. DDPMs generate high-quality samples from complex data distributions by defining an inverse process. Unlike generative adversarial networks (GANs), the latent space of diffusion models is less interpretable. In this work, we propose to generalize the denoising diffusion process into an Upsampling Diffusion Probabilistic Model (UDPM)
  arXiv  Detail & Related papers  (2023-05-25T17:25:14Z)
• DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion [137.8749239614528]
  We propose a new formulation of temporal action detection (TAD) with denoising diffusion, DiffTAD. Taking as input random temporal proposals, it can yield action proposals accurately given an untrimmed long video.
  arXiv  Detail & Related papers  (2023-03-27T00:40:52Z)
• Decoder Tuning: Efficient Language Understanding as Decoding [84.68266271483022]
  We present Decoder Tuning (DecT), which in contrast optimize task-specific decoder networks on the output side. By gradient-based optimization, DecT can be trained within several seconds and requires only one P query per sample. We conduct extensive natural language understanding experiments and show that DecT significantly outperforms state-of-the-art algorithms with a $200times$ speed-up.
  arXiv  Detail & Related papers  (2022-12-16T11:15:39Z)
• Post-training Quantization on Diffusion Models [14.167428759401703]
  Denoising diffusion (score-based) generative models have recently achieved significant accomplishments in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data into noise and a backward denoising process for sampling data from noise. Unfortunately, the generation process of current denoising diffusion models is notoriously slow due to the lengthy iterative noise estimations.
  arXiv  Detail & Related papers  (2022-11-28T19:33:39Z)
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models [14.734454356396157]
  We present a detailed overview of the diffusion process that is based on a differential equation. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models.
  arXiv  Detail & Related papers  (2022-08-11T13:55:12Z)
• ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech [63.780196620966905]
  We propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech. ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling. Our evaluation demonstrates that ProDiff needs only 2 iterations to synthesize high-fidelity mel-spectrograms. ProDiff enables a sampling speed of 24x faster than real-time on a single NVIDIA 2080Ti GPU.
  arXiv  Detail & Related papers  (2022-07-13T17:45:43Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models [57.20432226304683]
  Non-autoregressive (NAR) modeling has gained more and more attention in speech processing. We propose a novel end-to-end streaming NAR speech recognition system. We show that the proposed method improves online ASR recognition in low latency conditions.
  arXiv  Detail & Related papers  (2021-07-20T11:42:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.