Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement

• URL: http://arxiv.org/abs/2210.17287v3
• Date: Wed, 30 Aug 2023 10:18:25 GMT
• Title: Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement
• Authors: Ryosuke Sawata, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Takashi Shibuya, Shusuke Takahashi and Yuki Mitsufuji
• Abstract summary: We introduce a DNN-based generative refiner, Diffiner, to improve perceptual speech quality pre-processed by an SE method. Our method improved perceptual speech quality regardless of the preceding SE methods used.
• Score: 22.67630435329088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although deep neural network (DNN)-based speech enhancement (SE) methods outperform the previous non-DNN-based ones, they often degrade the perceptual quality of generated outputs. To tackle this problem, we introduce a DNN-based generative refiner, Diffiner, aiming to improve perceptual speech quality pre-processed by an SE method. We train a diffusion-based generative model by utilizing a dataset consisting of clean speech only. Then, our refiner effectively mixes clean parts newly generated via denoising diffusion restoration into the degraded and distorted parts caused by a preceding SE method, resulting in refined speech. Once our refiner is trained on a set of clean speech, it can be applied to various SE methods without additional training specialized for each SE module. Therefore, our refiner can be a versatile post-processing module w.r.t. SE methods and has high potential in terms of modularity. Experimental results show that our method improved perceptual speech quality regardless of the preceding SE methods used.

Related papers

• Diffusion-based Unsupervised Audio-visual Speech Enhancement [26.937216751657697]
  This paper proposes a new unsupervised audiovisual speech enhancement (AVSE) approach. It combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervisedgenerative AVSE method.
  arXiv  Detail & Related papers  (2024-10-04T12:22:54Z)
• Diffusion-based speech enhancement with a weighted generative-supervised learning loss [0.0]
  Diffusion-based generative models have recently gained attention in speech enhancement (SE) We propose augmenting the original diffusion training objective with a mean squared error (MSE) loss, measuring the discrepancy between estimated enhanced speech and ground-truth clean speech.
  arXiv  Detail & Related papers  (2023-09-19T09:13:35Z)
• Unsupervised speech enhancement with diffusion-based generative models [0.0]
  We introduce an alternative approach that operates in an unsupervised manner, leveraging the generative power of diffusion models. We develop a posterior sampling methodology for speech enhancement by combining the learnt clean speech prior with a noise model for speech signal inference. We show promising results compared to a recent variational auto-encoder (VAE)-based unsupervised approach and a state-of-the-art diffusion-based supervised method.
  arXiv  Detail & Related papers  (2023-09-19T09:11:31Z)
• Continuous Modeling of the Denoising Process for Speech Enhancement Based on Deep Learning [61.787485727134424]
  We use a state variable to indicate the denoising process. A UNet-like neural network learns to estimate every state variable sampled from the continuous denoising process. Experimental results indicate that preserving a small amount of noise in the clean target benefits speech enhancement.
  arXiv  Detail & Related papers  (2023-09-17T13:27:11Z)
• UnDiff: Unsupervised Voice Restoration with Unconditional Diffusion Model [1.0874597293913013]
  UnDiff is a diffusion probabilistic model capable of solving various speech inverse tasks. It can be adapted to different tasks including inversion degradation, neural vocoding, and source separation.
  arXiv  Detail & Related papers  (2023-06-01T14:22:55Z)
• VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation [88.49030739715701]
  This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation.
  arXiv  Detail & Related papers  (2023-03-15T02:16:39Z)
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models [14.734454356396157]
  We present a detailed overview of the diffusion process that is based on a differential equation. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models.
  arXiv  Detail & Related papers  (2022-08-11T13:55:12Z)
• An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks [112.1942546460814]
  We report the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM) Experiment results show that the prompt tuning technique achieves competitive performance in speech classification tasks with fewer trainable parameters than fine-tuning specialized downstream models.
  arXiv  Detail & Related papers  (2022-03-31T03:26:55Z)
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem [65.25725367771075]
  This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
  arXiv  Detail & Related papers  (2021-12-17T08:35:40Z)
• A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
  We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals. The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
  arXiv  Detail & Related papers  (2021-07-25T19:23:18Z)
• Rectified Meta-Learning from Noisy Labels for Robust Image-based Plant Disease Diagnosis [64.82680813427054]
  Plant diseases serve as one of main threats to food security and crop production. One popular approach is to transform this problem as a leaf image classification task, which can be addressed by the powerful convolutional neural networks (CNNs) We propose a novel framework that incorporates rectified meta-learning module into common CNN paradigm to train a noise-robust deep network without using extra supervision information.
  arXiv  Detail & Related papers  (2020-03-17T09:51:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.