Whisper Finetuning on Nepali Language

• URL: http://arxiv.org/abs/2411.12587v1
• Date: Tue, 19 Nov 2024 15:55:56 GMT
• Title: Whisper Finetuning on Nepali Language
• Authors: Sanjay Rijal, Shital Adhikari, Manish Dahal, Manish Awale, Vaghawan Ojha,
• Abstract summary: This research focuses on making an exhaustive and generalized dataset followed by fine-tuning OpenAI's Whisper models to improve transcription accuracy for the Nepali language. We leverage publicly available ASR datasets and self-recorded custom datasets with a diverse range of accents, dialects, and speaking styles further enriched through augmentation. Our approach outperforms Whisper's baseline models trained on Fleur's dataset, achieving WER reductions of up to 36.2% on the small and 23.8% on medium models.
• Score: 0.0
• License:
• Abstract: Despite the growing advancements in Automatic Speech Recognition (ASR) models, the development of robust models for underrepresented languages, such as Nepali, remains a challenge. This research focuses on making an exhaustive and generalized dataset followed by fine-tuning OpenAI's Whisper models of different sizes to improve transcription (speech-to-text) accuracy for the Nepali language. We leverage publicly available ASR datasets and self-recorded custom datasets with a diverse range of accents, dialects, and speaking styles further enriched through augmentation. Our experimental results demonstrate that fine-tuning Whisper models on our curated custom dataset substantially reduces the Word Error Rate (WER) across all model sizes attributed to larger data variations in terms of speaker's age, gender, and sentiment, acoustic environment, dialect, denser audio segments (15-30 seconds) that are more compatible with Whisper's input, and manual curation of audios and transcriptions. Notably, our approach outperforms Whisper's baseline models trained on Fleur's dataset, achieving WER reductions of up to 36.2% on the small and 23.8% on medium models. Furthermore, we show that data augmentation plays a significant role in enhancing model robustness. Our approach underlines the importance of dataset quality, variation, and augmentation in the adaptation of state-of-the-art models to underrepresented languages for developing accurate ASR systems.

Related papers

• Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model [0.0]
  OpenAI's Whisper Automated Speech Recognition model excels in generalizing across diverse datasets and domains. We propose a method to enhance transcription accuracy without explicit fine-tuning or altering model parameters.
  arXiv  Detail & Related papers  (2024-10-24T01:58:11Z)
• Enhancing Indonesian Automatic Speech Recognition: Evaluating Multilingual Models with Diverse Speech Variabilities [9.473861847584843]
  We present our research on state-of-the-art speech recognition models, namely Massively Multilingual Speech (MMS) and Whisper. We investigate the models' predictive ability to transcribe Indonesian speech data across different variability groups.
  arXiv  Detail & Related papers  (2024-10-11T14:07:07Z)
• Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text [22.19230427358921]
  It is worth researching how to improve the performance of Whisper on under-represented languages. We utilized easily accessible unpaired speech and text data and combined the language model GPT with Whisper on Kazakh. We achieved more than 10% absolute WER reduction in multiple experiments.
  arXiv  Detail & Related papers  (2024-08-10T13:39:13Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Automatic Speech Recognition Advancements for Indigenous Languages of the Americas [0.0]
  The Second Americas (Americas Natural Language Processing) Competition Track 1 of NeurIPS (Neural Information Processing Systems) 2022 proposed the task of training automatic speech recognition systems for five Indigenous languages: Quechua, Guarani, Bribri, Kotiria, and Wa'ikhana. We describe the fine-tuning of a state-of-the-art ASR model for each target language, using approximately 36.65 h of transcribed speech data from diverse sources enriched with data augmentation methods. We release our best models for each language, marking the first open ASR models for Wa'ikhana and Kotiria.
  arXiv  Detail & Related papers  (2024-04-12T10:12:38Z)
• Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer [59.57249127943914]
  We present a multilingual Audio-Visual Speech Recognition model incorporating several enhancements to improve performance and audio noise robustness. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets. Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%.
  arXiv  Detail & Related papers  (2024-03-14T01:16:32Z)
• Retrieval is Accurate Generation [99.24267226311157]
  We introduce a novel method that selects context-aware phrases from a collection of supporting documents. Our model achieves the best performance and the lowest latency among several retrieval-augmented baselines.
  arXiv  Detail & Related papers  (2024-02-27T14:16:19Z)
• Deepfake audio as a data augmentation technique for training automatic speech to text transcription models [55.2480439325792]
  We propose a framework that approaches data augmentation based on deepfake audio. A dataset produced by Indians (in English) was selected, ensuring the presence of a single accent.
  arXiv  Detail & Related papers  (2023-09-22T11:33:03Z)
• Visual Speech Recognition for Multiple Languages in the Wild [64.52593130370757]
  We show that designing better VSR models is equally important to using larger training sets. We propose the addition of prediction-based auxiliary tasks to a VSR model. We show that such model works for different languages and outperforms all previous methods trained on publicly available datasets by a large margin.
  arXiv  Detail & Related papers  (2022-02-26T07:21:00Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.