Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining

• URL: http://arxiv.org/abs/2501.03184v1
• Date: Mon, 06 Jan 2025 18:00:14 GMT
• Title: Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining
• Authors: Holger Severin Bovbjerg, Jan Østergaard, Jesper Jensen, Zheng-Hua Tan,
• Abstract summary: Target-Speaker Voice Activity Detection (TS-VAD) is the task of detecting the presence of speech from a known target-speaker in an audio frame. Deep neural network-based models have shown good performance in this task. We propose a causal, Self-Supervised Learning (SSL) pretraining framework to enhance TS-VAD performance in noisy conditions.
• Score: 21.26555178371168
• License:
• Abstract: Target-Speaker Voice Activity Detection (TS-VAD) is the task of detecting the presence of speech from a known target-speaker in an audio frame. Recently, deep neural network-based models have shown good performance in this task. However, training these models requires extensive labelled data, which is costly and time-consuming to obtain, particularly if generalization to unseen environments is crucial. To mitigate this, we propose a causal, Self-Supervised Learning (SSL) pretraining framework, called Denoising Autoregressive Predictive Coding (DN-APC), to enhance TS-VAD performance in noisy conditions. We also explore various speaker conditioning methods and evaluate their performance under different noisy conditions. Our experiments show that DN-APC improves performance in noisy conditions, with a general improvement of approx. 2% in both seen and unseen noise. Additionally, we find that FiLM conditioning provides the best overall performance. Representation analysis via tSNE plots reveals robust initial representations of speech and non-speech from pretraining. This underscores the effectiveness of SSL pretraining in improving the robustness and performance of TS-VAD models in noisy environments.

Related papers

• Towards Robust Transcription: Exploring Noise Injection Strategies for Training Data Augmentation [55.752737615873464]
  This study investigates the impact of white noise at various Signal-to-Noise Ratio (SNR) levels on state-of-the-art APT models. We hope this research provides valuable insights as preliminary work toward developing transcription models that maintain consistent performance across a range of acoustic conditions.
  arXiv  Detail & Related papers  (2024-10-18T02:31:36Z)
• Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance [42.90024643696503]
  We present an end-to-end learning solution to jointly optimise the models for audio enhancement. We consider four representative applications to evaluate our training paradigm.
  arXiv  Detail & Related papers  (2024-08-12T16:23:58Z)
• Robust Active Speaker Detection in Noisy Environments [29.785749048315616]
  We formulate a robust active speaker detection (rASD) problem in noisy environments. Existing ASD approaches leverage both audio and visual modalities, but non-speech sounds in the surrounding environment can negatively impact performance. We propose a novel framework that utilizes audio-visual speech separation as guidance to learn noise-free audio features.
  arXiv  Detail & Related papers  (2024-03-27T20:52:30Z)
• Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditions [0.0]
  We pretrain a long short-term memory (LSTM)-encoder using the autoregressive predictive coding framework. We also propose a denoising variant of APC, with the goal of improving the robustness of personalized VAD. Our experiments show that self-supervised pretraining not only improves performance in clean conditions, but also yields models which are more robust to adverse conditions.
  arXiv  Detail & Related papers  (2023-12-27T15:36:17Z)
• Continuous Modeling of the Denoising Process for Speech Enhancement Based on Deep Learning [61.787485727134424]
  We use a state variable to indicate the denoising process. A UNet-like neural network learns to estimate every state variable sampled from the continuous denoising process. Experimental results indicate that preserving a small amount of noise in the clean target benefits speech enhancement.
  arXiv  Detail & Related papers  (2023-09-17T13:27:11Z)
• Inference and Denoise: Causal Inference-based Neural Speech Enhancement [83.4641575757706]
  This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. The proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE.
  arXiv  Detail & Related papers  (2022-11-02T15:03:50Z)
• Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction [109.44933866397123]
  Noise robustness is essential for deploying automatic speech recognition systems in real-world environments. We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
  arXiv  Detail & Related papers  (2021-10-28T20:39:02Z)
• Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder [30.318947721658862]
  We propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters.
  arXiv  Detail & Related papers  (2021-02-17T11:40:42Z)
• Speech Enhancement for Wake-Up-Word detection in Voice Assistants [60.103753056973815]
  Keywords spotting and in particular Wake-Up-Word (WUW) detection is a very important task for voice assistants. This paper proposes a Speech Enhancement model adapted to the task of WUW detection. It aims at increasing the recognition rate and reducing the false alarms in the presence of these types of noises.
  arXiv  Detail & Related papers  (2021-01-29T18:44:05Z)
• CITISEN: A Deep Learning-Based Speech Signal-Processing Mobile Application [63.2243126704342]
  This study presents a deep learning-based speech signal-processing mobile application known as CITISEN. The CITISEN provides three functions: speech enhancement (SE), model adaptation (MA), and background noise conversion (BNC) Compared with the noisy speech signals, the enhanced speech signals achieved about 6% and 33% of improvements.
  arXiv  Detail & Related papers  (2020-08-21T02:04:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.