Multilingual Bottleneck Features for Improving ASR Performance of Code-Switched Speech in Under-Resourced Languages

• URL: http://arxiv.org/abs/2011.03118v1
• Date: Sat, 31 Oct 2020 18:51:42 GMT
• Title: Multilingual Bottleneck Features for Improving ASR Performance of Code-Switched Speech in Under-Resourced Languages
• Abstract summary: We explore the benefits of using multilingual bottleneck features (mBNF) in acoustic modelling for the automatic speech recognition of code-switched (CS) speech in African languages. We train a mBNF extractor using nine Southern Bantu languages that form part of the freely available multilingual NCHLT corpus. Our results show that the inclusion of the mBNF features leads to clear performance improvements over a baseline trained without the mBNFs for code-switched English-isiZulu, English-isiXa, English-Sesotho and English-Setswana speech.
• Score: 12.139300459657974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we explore the benefits of using multilingual bottleneck features (mBNF) in acoustic modelling for the automatic speech recognition of code-switched (CS) speech in African languages. The unavailability of annotated corpora in the languages of interest has always been a primary challenge when developing speech recognition systems for this severely under-resourced type of speech. Hence, it is worthwhile to investigate the potential of using speech corpora available for other better-resourced languages to improve speech recognition performance. To achieve this, we train a mBNF extractor using nine Southern Bantu languages that form part of the freely available multilingual NCHLT corpus. We append these mBNFs to the existing MFCCs, pitch features and i-vectors to train acoustic models for automatic speech recognition (ASR) in the target code-switched languages. Our results show that the inclusion of the mBNF features leads to clear performance improvements over a baseline trained without the mBNFs for code-switched English-isiZulu, English-isiXhosa, English-Sesotho and English-Setswana speech.

Related papers

• A2TTS: TTS for Low Resource Indian Languages [16.782842482372427]
  We present a speaker conditioned text-to-speech (TTS) system aimed at generating speech for unseen speakers.<n>Using a diffusion-based TTS architecture, a speaker encoder extracts embeddings from short reference audio samples to condition the DDPM decoder for multispeaker generation.<n>We employ a cross-attention based duration prediction mechanism that utilizes reference audio, enabling more accurate and speaker consistent timing.
  arXiv  Detail & Related papers  (2025-07-21T06:20:27Z)
• From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM [21.454870778985153]
  We extend an existing large language model (LLMs) to the speech modality via speech discretization and continued pre-training. The resulting open-source model, SPIRE, is able to transcribe and translate English speech input while maintaining TOWER's original performance on translation-related tasks.
  arXiv  Detail & Related papers  (2025-03-13T17:57:32Z)
• Zero-resource Speech Translation and Recognition with LLMs [38.11535502039386]
  We propose to leverage a multilingual Large Language Model (LLM) to perform ST and ASR in languages for which the model has never seen paired audio-text data. We achieve this by using a pre-trained multilingual speech encoder, a multilingual LLM, and a lightweight adaptation module that maps the audio representations to the token embedding space of the LLM.
  arXiv  Detail & Related papers  (2024-12-24T17:37:11Z)
• Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling [50.62091603179394]
  Whisper, one of the most advanced ASR models, handles 99 languages effectively. However, Whisper struggles with unseen languages, those not included in its pre-training. We propose methods that exploit these relationships to enhance ASR performance on unseen languages.
  arXiv  Detail & Related papers  (2024-12-21T04:05:43Z)
• LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliteration [19.403991814044424]
  We introduce a Language-Agnostic Multilingual ASR pipeline through orthography Unification and language-specific Transliteration (LAMA-UT) LAMA-UT operates without any language-specific modules while matching the performance of state-of-the-art models trained on a minimal amount of data. Our pipeline achieves a relative error reduction rate of 45% when compared to Whisper and performs comparably to MMS.
  arXiv  Detail & Related papers  (2024-12-19T10:39:08Z)
• Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data [84.01401439030265]
  Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) We present a simple yet effective automatic process for creating speech-text pair data. Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data.
  arXiv  Detail & Related papers  (2024-09-30T07:01:21Z)
• Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models [13.855545744177586]
  This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations.<n>Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data.<n>The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.
  arXiv  Detail & Related papers  (2024-09-17T09:04:03Z)
• Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection [49.27067541740956]
  Speech Emotion Recognition (SER) is a crucial component in developing general-purpose AI agents capable of natural human-computer interaction. Building robust multilingual SER systems remains challenging due to the scarcity of labeled data in languages other than English and Chinese. We propose an approach to enhance SER performance in low SER resource languages by leveraging data from high-resource languages.
  arXiv  Detail & Related papers  (2024-09-17T08:36:45Z)
• Large Language Models for Dysfluency Detection in Stuttered Speech [16.812800649507302]
  Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech.
  arXiv  Detail & Related papers  (2024-06-16T17:51:22Z)
• AudioPaLM: A Large Language Model That Can Speak and Listen [79.44757696533709]
  We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models. It can process and generate text and speech with applications including speech recognition and speech-to-speech translation.
  arXiv  Detail & Related papers  (2023-06-22T14:37:54Z)
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling [92.55131711064935]
  We propose a cross-lingual neural language model, VALL-E X, for cross-lingual speech synthesis. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. It can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment.
  arXiv  Detail & Related papers  (2023-03-07T14:31:55Z)
• Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition [62.997667081978825]
  Code-Switching (CS) is referred to the phenomenon of alternately using words and phrases from different languages. We propose a simple yet effective data augmentation in which audio and corresponding labels of different source languages are transcribed. We show that this augmentation can even improve the model's performance on inter-sentential language switches not seen during training by 5,03% WER.
  arXiv  Detail & Related papers  (2022-10-17T12:15:57Z)
• Zero-Shot Cross-lingual Aphasia Detection using Automatic Speech Recognition [3.2631198264090746]
  Aphasia is a common speech and language disorder, typically caused by a brain injury or a stroke, that affects millions of people worldwide. We propose an end-to-end pipeline using pre-trained Automatic Speech Recognition (ASR) models that share cross-lingual speech representations.
  arXiv  Detail & Related papers  (2022-04-01T14:05:02Z)
• Voice Conversion Can Improve ASR in Very Low-Resource Settings [32.170748231414365]
  We study whether a VC system can be used cross-lingually to improve low-resource speech recognition. We combine several recent techniques to design and train a practical VC system in English. We find that when using a sensible amount of augmented data, speech recognition performance is improved in all four low-resource languages considered.
  arXiv  Detail & Related papers  (2021-11-04T07:57:00Z)
• Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity [81.51206991542242]
  Cross-lingual transfer offers a compelling way to help bridge this digital divide. Current cross-lingual algorithms have shown success in text-based tasks and speech-related tasks over some low-resource languages. We propose a language similarity approach that can efficiently identify acoustic cross-lingual transfer pairs across hundreds of languages.
  arXiv  Detail & Related papers  (2021-11-02T01:55:17Z)
• Language ID Prediction from Speech Using Self-Attentive Pooling and 1D-Convolutions [0.0]
  This memo describes NTR-TSU submission for SIGTYP 2021 Shared Task on predicting language IDs from speech. For many low-resource and endangered languages, only single-speaker recordings may be available, demanding a need for domain and speaker-invariant language ID systems. We show that a convolutional neural network with a Self-Attentive Pooling layer shows promising results for the language identification task.
  arXiv  Detail & Related papers  (2021-04-24T16:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.