Adaptation of Whisper models to child speech recognition

• URL: http://arxiv.org/abs/2307.13008v1
• Date: Mon, 24 Jul 2023 12:54:45 GMT
• Title: Adaptation of Whisper models to child speech recognition
• Authors: Rishabh Jain and Andrei Barcovschi and Mariam Yiwere and Peter Corcoran and Horia Cucu
• Abstract summary: We show that finetuning Whisper on child speech yields significant improvements in ASR performance on child speech. utilizing self-supervised Wav2vec2 models that have been finetuned on child speech outperforms Whisper finetuning.
• Score: 3.2548794659022398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic Speech Recognition (ASR) systems often struggle with transcribing child speech due to the lack of large child speech datasets required to accurately train child-friendly ASR models. However, there are huge amounts of annotated adult speech datasets which were used to create multilingual ASR models, such as Whisper. Our work aims to explore whether such models can be adapted to child speech to improve ASR for children. In addition, we compare Whisper child-adaptations with finetuned self-supervised models, such as wav2vec2. We demonstrate that finetuning Whisper on child speech yields significant improvements in ASR performance on child speech, compared to non finetuned Whisper models. Additionally, utilizing self-supervised Wav2vec2 models that have been finetuned on child speech outperforms Whisper finetuning.

Related papers

• Evaluation of state-of-the-art ASR Models in Child-Adult Interactions [27.30130353688078]
  Speech foundation models show a noticeable performance drop (15-20% absolute WER) for child speech compared to adult speech in the conversational setting. We employ LoRA on the best performing zero shot model (whisper-large) to probe the effectiveness of fine-tuning in a low resource setting.
  arXiv  Detail & Related papers  (2024-09-24T14:42:37Z)
• Improving child speech recognition with augmented child-like speech [20.709414063132627]
  Cross-lingual child-to-child voice conversion significantly improved child ASR performance. State-of-the-art ASRs show suboptimal performance for child speech.
  arXiv  Detail & Related papers  (2024-06-12T08:56:46Z)
• Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer [59.57249127943914]
  We present a multilingual Audio-Visual Speech Recognition model incorporating several enhancements to improve performance and audio noise robustness. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets. Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%.
  arXiv  Detail & Related papers  (2024-03-14T01:16:32Z)
• Pheme: Efficient and Conversational Speech Generation [52.34331755341856]
  We introduce the Pheme model series that offers compact yet high-performing conversational TTS models. It can be trained efficiently on smaller-scale conversational data, cutting data demands by more than 10x but still matching the quality of the autoregressive TTS models.
  arXiv  Detail & Related papers  (2024-01-05T14:47:20Z)
• A comparative analysis between Conformer-Transducer, Whisper, and wav2vec2 for improving the child speech recognition [2.965450563218781]
  We show that finetuning Conformer-transducer models on child speech yields significant improvements in ASR performance on child speech. We also show Whisper and wav2vec2 adaptation on different child speech datasets.
  arXiv  Detail & Related papers  (2023-11-07T19:32:48Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Automatic Speech Recognition of Non-Native Child Speech for Language Learning Applications [18.849741353784328]
  We assess the performance of two state-of-the-art ASR systems, Wav2Vec2.0 and Whisper AI. We evaluate their performance on read and extemporaneous speech of native and non-native Dutch children.
  arXiv  Detail & Related papers  (2023-06-29T06:14:26Z)
• AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR [79.21857972093332]
  We present AVFormer, a method for augmenting audio-only models with visual information, at the same time performing lightweight domain adaptation. We show that these can be trained on a small amount of weakly labelled video data with minimum additional training time and parameters. We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively.
  arXiv  Detail & Related papers  (2023-03-29T07:24:28Z)
• Bridging Speech and Textual Pre-trained Models with Unsupervised ASR [70.61449720963235]
  This work proposes a simple yet efficient unsupervised paradigm that connects speech and textual pre-trained models. We show that unsupervised automatic speech recognition (ASR) can improve the representations from speech self-supervised models. Notably, on spoken question answering, we reach the state-of-the-art result over the challenging NMSQA benchmark.
  arXiv  Detail & Related papers  (2022-11-06T04:50:37Z)
• Transfer Learning for Robust Low-Resource Children's Speech ASR with Transformers and Source-Filter Warping [11.584388304271029]
  We propose a data augmentation technique based on the source-filter model of speech to close the domain gap between adult and children's speech. Using this augmentation strategy, we apply transfer learning on a Transformer model pre-trained on adult data. This model follows the recently introduced XLS-R architecture, a wav2vec 2.0 model pre-trained on several cross-lingual adult speech corpora.
  arXiv  Detail & Related papers  (2022-06-19T12:57:47Z)
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation [51.37980448183019]
  We propose Audio ALBERT, a lite version of the self-supervised speech representation model. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.
  arXiv  Detail & Related papers  (2020-05-18T10:42:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.