Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

• URL: http://arxiv.org/abs/2606.21215v1
• Date: Fri, 19 Jun 2026 08:32:07 GMT
• Title: Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
• Abstract summary: We present the first systematic study across 10 non-verbal vocalizations (NVVs)<n>We propose a framework combining Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing.<n>A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap.
• Score: 70.31217233606066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.

Related papers

• Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations [13.518435276876973]
  We develop an NVV-aware DiTAR system that models continuous speech latents, encodes the 16 target NVV categories as dedicated to- kens, and adapts stop prediction to distinguish mid-utterance vocalizations from utterance boundaries.<n>The system achieves an official weighted bilingual score of 62.786, ranking first in Mandarin, second in English, and first overall among participating systems in Track 2 of the ISCSLP 2026 NVVSpeech Challenge.
  arXiv  Detail & Related papers  (2026-09-13T01:48:37Z)
• From Silent Signals to Natural Language: A Dual-Stage Transformer-LLM Approach [0.0]
  We propose an automatic speech recognition framework that combines a transformer-based acoustic model with a large language model (LLM) for post-processing.<n> Experimental results show a 16% relative and 6% absolute reduction in word error rate (WER) over a 36% baseline, demonstrating substantial improvements in intelligibility for silent speech interfaces.
  arXiv  Detail & Related papers  (2025-09-02T16:13:29Z)
• Investigation of Data Augmentation Techniques for Disordered Speech Recognition [69.50670302435174]
  This paper investigates a set of data augmentation techniques for disordered speech recognition. Both normal and disordered speech were exploited in the augmentation process. The final speaker adapted system constructed using the UASpeech corpus and the best augmentation approach based on speed perturbation produced up to 2.92% absolute word error rate (WER)
  arXiv  Detail & Related papers  (2022-01-14T17:09:22Z)
• Private Language Model Adaptation for Speech Recognition [15.726921748859393]
  Speech model adaptation is crucial to handle the discrepancy between server-side proxy training data and actual data received on users' local devices. We introduce an efficient approach on continuously adapting neural network language models (NNLMs) on private devices with applications on automatic speech recognition.
  arXiv  Detail & Related papers  (2021-09-28T00:15:43Z)
• Injecting Text in Self-Supervised Speech Pretraining [33.676479965610774]
  We propose to jointly learn representations during pretraining from two different modalities: speech and text. tts4pretrain complements the power of contrastive learning in self-supervision. We demonstrate Word Error Rate (WER) reductions of 10% relative on the well-benchmarked, Librispeech task.
  arXiv  Detail & Related papers  (2021-08-27T11:36:40Z)
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
  One-shot voice conversion can be effectively achieved by speech representation disentanglement. We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
  arXiv  Detail & Related papers  (2021-06-18T13:50:38Z)
• End-to-End Spoken Language Understanding for Generalized Voice Assistants [15.241812584273886]
  We present our approach to developing an E2E model for generalized speech recognition in commercial voice assistants (VAs) We propose a fully differentiable, transformer-based, hierarchical system that can be pretrained at both the ASR and NLU levels. This is then fine-tuned on both transcription and semantic classification losses to handle a diverse set of intent and argument combinations.
  arXiv  Detail & Related papers  (2021-06-16T17:56:47Z)
• Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition [62.94773371761236]
  We consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate. We propose a method of dynamic acoustic unit augmentation based on the BPE-dropout technique. Our monolingual Turkish Conformer established a competitive result with 22.2% character error rate (CER) and 38.9% word error rate (WER)
  arXiv  Detail & Related papers  (2021-03-12T10:10:13Z)
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data [54.733889961024445]
  We propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data. We evaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus.
  arXiv  Detail & Related papers  (2021-01-19T12:53:43Z)
• Data augmentation using prosody and false starts to recognize non-native children's speech [12.911954427107977]
  This paper describes AaltoASR's speech recognition system for the INTERSPEECH 2020 shared task on Automatic Speech Recognition. The task is to recognize non-native speech from children of various age groups given a limited amount of speech.
  arXiv  Detail & Related papers  (2020-08-29T05:32:32Z)
• Continuous Speech Separation with Conformer [60.938212082732775]
  We use transformer and conformer in lieu of recurrent neural networks in the separation system. We believe capturing global information with the self-attention based method is crucial for the speech separation.
  arXiv  Detail & Related papers  (2020-08-13T09:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.