Related papers: Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments

Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments

URL: http://arxiv.org/abs/2410.05423v1
Date: Mon, 7 Oct 2024 18:39:59 GMT
Title: Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments
Authors: Sagarika Alavilli, Annesya Banerjee, Gasser Elbanna, Annika Magaro,
Abstract summary: We develop a transformer-based model that jointly performs speech recognition and speaker identification. We show that the joint model performs comparably to Whisper under clean conditions. Our results suggest that integrating voice representations with speech recognition can lead to more robust models under adversarial conditions.
Score: 0.2916558661202724
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Current state-of-the-art speech recognition models are trained to map acoustic signals into sub-lexical units. While these models demonstrate superior performance, they remain vulnerable to out-of-distribution conditions such as background noise and speech augmentations. In this work, we hypothesize that incorporating speaker representations during speech recognition can enhance model robustness to noise. We developed a transformer-based model that jointly performs speech recognition and speaker identification. Our model utilizes speech embeddings from Whisper and speaker embeddings from ECAPA-TDNN, which are processed jointly to perform both tasks. We show that the joint model performs comparably to Whisper under clean conditions. Notably, the joint model outperforms Whisper in high-noise environments, such as with 8-speaker babble background noise. Furthermore, our joint model excels in handling highly augmented speech, including sine-wave and noise-vocoded speech. Overall, these results suggest that integrating voice representations with speech recognition can lead to more robust models under adversarial conditions.