Enhancing Speaker Verification with Whispered Speech via Post-Processing

• URL: http://arxiv.org/abs/2604.20229v1
• Date: Wed, 22 Apr 2026 06:23:40 GMT
• Title: Enhancing Speaker Verification with Whispered Speech via Post-Processing
• Abstract summary: We propose a model with a training recipe to obtain more robust representations against whispered speech hindrances.<n>The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone.<n>We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech.
• Score: 1.8849814100256281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speaker verification is a task of confirming an individual's identity through the analysis of their voice. Whispered speech differs from phonated speech in acoustic characteristics, which degrades the performance of speaker verification systems in real-life scenarios, including avoiding fully phonated speech to protect privacy, disrupt others, or when the lack of full vocalization is dictated by a disease. In this paper we propose a model with a training recipe to obtain more robust representations against whispered speech hindrances. The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone, optimized jointly using cosine similarity--based classification and triplet loss. We gain relative improvement of 22.26\% compared to the baseline (baseline 6.77\% vs ours 5.27\%) in normal vs whispered speech trials, achieving AUC of 98.16\%. In tests comparing whispered to whispered, our model attains an EER of 1.88\% with AUC equal to 99.73\%, which represents a 15\% relative enhancement over the prior leading ReDimNet-B2. We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech. Additionally, we evaluate how these models perform under noisy audios, obtaining that generally the same relative level of noise degrades the performance of speaker verification more significantly on whispered speech than on normal speech.

Related papers

• HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding [54.365959989048406]
  Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments.<n>But their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear.<n>We introduce HEAR, a benchmark diagnosing the foundational capabilities of speaker-attributed reasoning.<n>We present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives.
  arXiv  Detail & Related papers  (2026-08-29T08:01:20Z)
• Targeted Speaker Poisoning Framework in Zero-Shot Text-to-Speech [42.343691065979215]
  Zero-shot Text-to-Speech (TTS) voice cloning poses severe privacy risks.<n>We formalize this task as Speech Generation Speaker Poisoning (SGSP)<n>We modify trained models to prevent the generation of specific identities while preserving utility for other speakers.
  arXiv  Detail & Related papers  (2026-03-08T09:29:55Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition [52.11964238935099]
  An audio-visual multi-channel speech separation, dereverberation and recognition approach is proposed in this paper. Video input is consistently demonstrated in mask-based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset.
  arXiv  Detail & Related papers  (2023-07-06T10:50:46Z)
• ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations [12.20522794248598]
  We propose a zero-shot voice conversion method using speech representations trained with self-supervised learning. We develop a multi-task model to decompose a speech utterance into features such as linguistic content, speaker characteristics, and speaking style. Next, we develop a synthesis model with pitch and duration predictors that can effectively reconstruct the speech signal from its representation.
  arXiv  Detail & Related papers  (2023-02-16T08:10:41Z)
• Self-Supervised Speech Representations Preserve Speech Characteristics while Anonymizing Voices [15.136348385992047]
  We train several voice conversion models using self-supervised speech representations. Converted voices retain a low word error rate within 1% of the original voice. Experiments on dysarthric speech data show that speech features relevant to articulation, prosody, phonation and phonology can be extracted from anonymized voices.
  arXiv  Detail & Related papers  (2022-04-04T17:48:01Z)
• Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation [59.41186714127256]
  Dysarthric speech reconstruction (DSR) aims to improve the quality of dysarthric speech. Speaker encoder (SE) optimized for speaker verification has been explored to control the speaker identity. We propose a novel multi-task learning strategy, i.e., adversarial speaker adaptation (ASA)
  arXiv  Detail & Related papers  (2022-02-18T08:59:36Z)
• Robust Self-Supervised Audio-Visual Speech Recognition [29.526786921769613]
  We present a self-supervised audio-visual speech recognition framework built upon Audio-Visual HuBERT (AV-HuBERT) On the largest available AVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by 50% (28.0% vs. 14.1%) using less than 10% of labeled data. Our approach reduces the WER of an audio-based model by over 75% (25.8% vs. 5.8%) on average.
  arXiv  Detail & Related papers  (2022-01-05T18:50:50Z)
• PL-EESR: Perceptual Loss Based END-TO-END Robust Speaker Representation Extraction [90.55375210094995]
  Speech enhancement aims to improve the perceptual quality of the speech signal by suppression of the background noise. We propose an end-to-end deep learning framework, dubbed PL-EESR, for robust speaker representation extraction.
  arXiv  Detail & Related papers  (2021-10-03T07:05:29Z)
• Learning Explicit Prosody Models and Deep Speaker Embeddings for Atypical Voice Conversion [60.808838088376675]
  We propose a VC system with explicit prosodic modelling and deep speaker embedding learning. A prosody corrector takes in phoneme embeddings to infer typical phoneme duration and pitch values. A conversion model takes phoneme embeddings and typical prosody features as inputs to generate the converted speech.
  arXiv  Detail & Related papers  (2020-11-03T13:08:53Z)
• Robust Speaker Recognition Using Speech Enhancement And Attention Model [37.33388614967888]
  Instead of individually processing speech enhancement and speaker recognition, the two modules are integrated into one framework by a joint optimisation using deep neural networks. To increase robustness against noise, a multi-stage attention mechanism is employed to highlight the speaker related features learned from context information in time and frequency domain. The obtained results show that the proposed approach using speech enhancement and multi-stage attention models outperforms two strong baselines not using them in most acoustic conditions in our experiments.
  arXiv  Detail & Related papers  (2020-01-14T20:03:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.