Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion
- URL: http://arxiv.org/abs/2306.05708v2
- Date: Mon, 12 Jun 2023 06:12:41 GMT
- Title: Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion
- Authors: Haogeng Liu, Tao Wang, Jie Cao, Ran He, Jianhua Tao
- Abstract summary: This paper proposes a linear diffusion model (LinDiff) based on an ordinary differential equation to simultaneously reach fast inference and high sample quality.
To reduce computational complexity, LinDiff employs a patch-based processing approach that partitions the input signal into small patches.
Our model can synthesize speech of a quality comparable to that of autoregressive models with faster synthesis speed.
- Score: 85.54515118077825
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Denoising Diffusion Probabilistic Models have shown extraordinary ability on
various generative tasks. However, their slow inference speed renders them
impractical in speech synthesis. This paper proposes a linear diffusion model
(LinDiff) based on an ordinary differential equation to simultaneously reach
fast inference and high sample quality. Firstly, we employ linear interpolation
between the target and noise to design a diffusion sequence for training, while
previously the diffusion path that links the noise and target is a curved
segment. When decreasing the number of sampling steps (i.e., the number of line
segments used to fit the path), the ease of fitting straight lines compared to
curves allows us to generate higher quality samples from a random noise with
fewer iterations. Secondly, to reduce computational complexity and achieve
effective global modeling of noisy speech, LinDiff employs a patch-based
processing approach that partitions the input signal into small patches. The
patch-wise token leverages Transformer architecture for effective modeling of
global information. Adversarial training is used to further improve the sample
quality with decreased sampling steps. We test proposed method with speech
synthesis conditioned on acoustic feature (Mel-spectrograms). Experimental
results verify that our model can synthesize high-quality speech even with only
one diffusion step. Both subjective and objective evaluations demonstrate that
our model can synthesize speech of a quality comparable to that of
autoregressive models with faster synthesis speed (3 diffusion steps).
Related papers
- Score-based Generative Models with Adaptive Momentum [40.84399531998246]
We propose an adaptive momentum sampling method to accelerate the transforming process.
We show that our method can produce more faithful images/graphs in small sampling steps with 2 to 5 times speed up.
arXiv Detail & Related papers (2024-05-22T15:20:27Z) - SpecDiff-GAN: A Spectrally-Shaped Noise Diffusion GAN for Speech and
Music Synthesis [0.0]
We introduce SpecDiff-GAN, a neural vocoder based on HiFi-GAN.
We show the merits of our proposed model for speech and music synthesis on several datasets.
arXiv Detail & Related papers (2024-01-30T09:17:57Z) - Adversarial Training of Denoising Diffusion Model Using Dual
Discriminators for High-Fidelity Multi-Speaker TTS [0.0]
The diffusion model is capable of generating high-quality data through a probabilistic approach.
It suffers from the drawback of slow generation speed due to the requirement of a large number of time steps.
We propose a speech synthesis model with two discriminators: a diffusion discriminator for learning the distribution of the reverse process and a spectrogram discriminator for learning the distribution of the generated data.
arXiv Detail & Related papers (2023-08-03T07:22:04Z) - ProDiff: Progressive Fast Diffusion Model For High-Quality
Text-to-Speech [63.780196620966905]
We propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech.
ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling.
Our evaluation demonstrates that ProDiff needs only 2 iterations to synthesize high-fidelity mel-spectrograms.
ProDiff enables a sampling speed of 24x faster than real-time on a single NVIDIA 2080Ti GPU.
arXiv Detail & Related papers (2022-07-13T17:45:43Z) - FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech
Synthesis [90.3069686272524]
This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis.
FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies.
Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms.
arXiv Detail & Related papers (2022-04-21T07:49:09Z) - A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals.
The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
arXiv Detail & Related papers (2021-07-25T19:23:18Z) - DiffSinger: Diffusion Acoustic Model for Singing Voice Synthesis [53.19363127760314]
DiffSinger is a parameterized Markov chain which iteratively converts the noise into mel-spectrogram conditioned on the music score.
The evaluations conducted on the Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work with a notable margin.
arXiv Detail & Related papers (2021-05-06T05:21:42Z) - Temporal-Spatial Neural Filter: Direction Informed End-to-End
Multi-channel Target Speech Separation [66.46123655365113]
Target speech separation refers to extracting the target speaker's speech from mixed signals.
Two main challenges are the complex acoustic environment and the real-time processing requirement.
We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
arXiv Detail & Related papers (2020-01-02T11:12:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.