Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance

• URL: http://arxiv.org/abs/2408.06264v1
• Date: Mon, 12 Aug 2024 16:23:58 GMT
• Title: Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance
• Authors: Manuel Milling, Shuo Liu, Andreas Triantafyllopoulos, Ilhan Aslan, Björn W. Schuller,
• Abstract summary: We present an end-to-end learning solution to jointly optimise the models for audio enhancement. We consider four representative applications to evaluate our training paradigm.
• Score: 42.90024643696503
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Neural network models for audio tasks, such as automatic speech recognition (ASR) and acoustic scene classification (ASC), are susceptible to noise contamination for real-life applications. To improve audio quality, an enhancement module, which can be developed independently, is explicitly used at the front-end of the target audio applications. In this paper, we present an end-to-end learning solution to jointly optimise the models for audio enhancement (AE) and the subsequent applications. To guide the optimisation of the AE module towards a target application, and especially to overcome difficult samples, we make use of the sample-wise performance measure as an indication of sample importance. In experiments, we consider four representative applications to evaluate our training paradigm, i.e., ASR, speech command recognition (SCR), speech emotion recognition (SER), and ASC. These applications are associated with speech and non-speech tasks concerning semantic and non-semantic features, transient and global information, and the experimental results indicate that our proposed approach can considerably boost the noise robustness of the models, especially at low signal-to-noise ratios (SNRs), for a wide range of computer audition tasks in everyday-life noisy environments.

Related papers

• Robust Active Speaker Detection in Noisy Environments [29.785749048315616]
  We formulate a robust active speaker detection (rASD) problem in noisy environments. Existing ASD approaches leverage both audio and visual modalities, but non-speech sounds in the surrounding environment can negatively impact performance. We propose a novel framework that utilizes audio-visual speech separation as guidance to learn noise-free audio features.
  arXiv  Detail & Related papers  (2024-03-27T20:52:30Z)
• DiffSED: Sound Event Detection with Denoising Diffusion [70.18051526555512]
  We reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process. During training, our model learns to reverse the noising process by converting noisy latent queries to the groundtruth versions.
  arXiv  Detail & Related papers  (2023-08-14T17:29:41Z)
• A General Framework for Learning Procedural Audio Models of Environmental Sounds [7.478290484139404]
  This paper introduces the Procedural (audio) Variational autoEncoder (ProVE) framework as a general approach to learning Procedural Audio PA models. We show that ProVE models outperform both classical PA models and an adversarial-based approach in terms of sound fidelity.
  arXiv  Detail & Related papers  (2023-03-04T12:12:26Z)
• Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning [25.743503223389784]
  We propose a reinforcement learning (RL) based framework called MSRL. We customize a reward function directly related to task-specific metrics. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions.
  arXiv  Detail & Related papers  (2022-12-10T14:01:54Z)
• Inference and Denoise: Causal Inference-based Neural Speech Enhancement [83.4641575757706]
  This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. The proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE.
  arXiv  Detail & Related papers  (2022-11-02T15:03:50Z)
• Towards Intelligibility-Oriented Audio-Visual Speech Enhancement [8.19144665585397]
  We present a fully convolutional AV SE model that uses a modified short-time objective intelligibility (STOI) metric as a training cost function. Our proposed I-O AV SE framework outperforms audio-only (AO) and AV models trained with conventional distance-based loss functions.
  arXiv  Detail & Related papers  (2021-11-18T11:47:37Z)
• Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction [109.44933866397123]
  Noise robustness is essential for deploying automatic speech recognition systems in real-world environments. We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
  arXiv  Detail & Related papers  (2021-10-28T20:39:02Z)
• Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder [30.318947721658862]
  We propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters.
  arXiv  Detail & Related papers  (2021-02-17T11:40:42Z)
• CITISEN: A Deep Learning-Based Speech Signal-Processing Mobile Application [63.2243126704342]
  This study presents a deep learning-based speech signal-processing mobile application known as CITISEN. The CITISEN provides three functions: speech enhancement (SE), model adaptation (MA), and background noise conversion (BNC) Compared with the noisy speech signals, the enhanced speech signals achieved about 6% and 33% of improvements.
  arXiv  Detail & Related papers  (2020-08-21T02:04:12Z)
• Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention [70.82604384963679]
  This paper investigates a self-adaptation method for speech enhancement using auxiliary speaker-aware features. We extract a speaker representation used for adaptation directly from the test utterance.
  arXiv  Detail & Related papers  (2020-02-14T05:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.