Diffusion Conditional Expectation Model for Efficient and Robust Target
Speech Extraction
- URL: http://arxiv.org/abs/2309.13874v1
- Date: Mon, 25 Sep 2023 04:58:38 GMT
- Title: Diffusion Conditional Expectation Model for Efficient and Robust Target
Speech Extraction
- Authors: Leying Zhang, Yao Qian, Linfeng Yu, Heming Wang, Xinkai Wang, Hemin
Yang, Long Zhou, Shujie Liu, Yanmin Qian, Michael Zeng
- Abstract summary: We propose an efficient generative approach named Conditional Diffusion Expectation Model (DCEM) for Target Speech Extraction (TSE)
It can handle multi- and single-speaker scenarios in both noisy and clean conditions.
Our method outperforms conventional methods in terms of both intrusive and non-intrusive metrics.
- Score: 73.43534824551236
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Target Speech Extraction (TSE) is a crucial task in speech processing that
focuses on isolating the clean speech of a specific speaker from complex
mixtures. While discriminative methods are commonly used for TSE, they can
introduce distortion in terms of speech perception quality. On the other hand,
generative approaches, particularly diffusion-based methods, can enhance speech
quality perceptually but suffer from slower inference speed. We propose an
efficient generative approach named Diffusion Conditional Expectation Model
(DCEM) for TSE. It can handle multi- and single-speaker scenarios in both noisy
and clean conditions. Additionally, we introduce Regenerate-DCEM (R-DCEM) that
can regenerate and optimize speech quality based on pre-processed speech from a
discriminative model. Our method outperforms conventional methods in terms of
both intrusive and non-intrusive metrics and demonstrates notable strengths in
inference efficiency and robustness to unseen tasks. Audio examples are
available online (https://vivian556123.github.io/dcem).
Related papers
- High-Fidelity Speech Synthesis with Minimal Supervision: All Using
Diffusion Models [56.00939852727501]
Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations.
Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
arXiv Detail & Related papers (2023-09-27T09:27:03Z) - Diffusion-based speech enhancement with a weighted generative-supervised
learning loss [0.0]
Diffusion-based generative models have recently gained attention in speech enhancement (SE)
We propose augmenting the original diffusion training objective with a mean squared error (MSE) loss, measuring the discrepancy between estimated enhanced speech and ground-truth clean speech.
arXiv Detail & Related papers (2023-09-19T09:13:35Z) - Unsupervised speech enhancement with diffusion-based generative models [0.0]
We introduce an alternative approach that operates in an unsupervised manner, leveraging the generative power of diffusion models.
We develop a posterior sampling methodology for speech enhancement by combining the learnt clean speech prior with a noise model for speech signal inference.
We show promising results compared to a recent variational auto-encoder (VAE)-based unsupervised approach and a state-of-the-art diffusion-based supervised method.
arXiv Detail & Related papers (2023-09-19T09:11:31Z) - An Efficient Membership Inference Attack for the Diffusion Model by
Proximal Initialization [58.88327181933151]
In this paper, we propose an efficient query-based membership inference attack (MIA)
Experimental results indicate that the proposed method can achieve competitive performance with only two queries on both discrete-time and continuous-time diffusion models.
To the best of our knowledge, this work is the first to study the robustness of diffusion models to MIA in the text-to-speech task.
arXiv Detail & Related papers (2023-05-26T16:38:48Z) - Diffiner: A Versatile Diffusion-based Generative Refiner for Speech
Enhancement [22.67630435329088]
We introduce a DNN-based generative refiner, Diffiner, to improve perceptual speech quality pre-processed by an SE method.
Our method improved perceptual speech quality regardless of the preceding SE methods used.
arXiv Detail & Related papers (2022-10-27T10:46:32Z) - Speech Enhancement and Dereverberation with Diffusion-based Generative
Models [14.734454356396157]
We present a detailed overview of the diffusion process that is based on a differential equation.
We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates.
In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models.
arXiv Detail & Related papers (2022-08-11T13:55:12Z) - Conditional Diffusion Probabilistic Model for Speech Enhancement [101.4893074984667]
We propose a novel speech enhancement algorithm that incorporates characteristics of the observed noisy speech signal into the diffusion and reverse processes.
In our experiments, we demonstrate strong performance of the proposed approach compared to representative generative models.
arXiv Detail & Related papers (2022-02-10T18:58:01Z) - Discretization and Re-synthesis: an alternative method to solve the
Cocktail Party Problem [65.25725367771075]
This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem.
Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols.
By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
arXiv Detail & Related papers (2021-12-17T08:35:40Z) - A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals.
The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
arXiv Detail & Related papers (2021-07-25T19:23:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.