Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting

• URL: http://arxiv.org/abs/2406.12611v1
• Date: Tue, 18 Jun 2024 13:38:58 GMT
• Title: Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting
• Authors: Yosuke Kashiwagi, Hayato Futami, Emiru Tsunoo, Siddhant Arora, Shinji Watanabe,
• Abstract summary: We introduce an encoder prompting technique within the self-conditioned CTC framework, enabling language-specific adaptation of the CTC model in a zero-shot manner. Our method has shown to significantly reduce errors by 28% on average and by 41% on low-resource languages.
• Score: 45.161909551392085
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end multilingual speech recognition models handle multiple languages through a single model, often incorporating language identification to automatically detect the language of incoming speech. Since the common scenario is where the language is already known, these models can perform as language-specific by using language information as prompts, which is particularly beneficial for attention-based encoder-decoder architectures. However, the Connectionist Temporal Classification (CTC) approach, which enhances recognition via joint decoding and multi-task training, does not normally incorporate language prompts due to its conditionally independent output tokens. To overcome this, we introduce an encoder prompting technique within the self-conditioned CTC framework, enabling language-specific adaptation of the CTC model in a zero-shot manner. Our method has shown to significantly reduce errors by 28% on average and by 41% on low-resource languages.

Related papers

• Leveraging Language ID to Calculate Intermediate CTC Loss for Enhanced Code-Switching Speech Recognition [5.3545957730615905]
  We introduce language identification information into the middle layer of the ASR model's encoder. We aim to generate acoustic features that imply language distinctions in a more implicit way, reducing the model's confusion when dealing with language switching.
  arXiv  Detail & Related papers  (2023-12-15T07:46:35Z)
• Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching [65.74653592668743]
  Finetuning self-supervised multilingual representations reduces absolute word error rates by up to 20%. In circumstances with limited training data finetuning self-supervised representations is a better performing and viable solution.
  arXiv  Detail & Related papers  (2023-11-25T17:05:21Z)
• Adapting the adapters for code-switching in multilingual ASR [10.316724084739892]
  Large pre-trained multilingual speech models have shown potential in scaling Automatic Speech Recognition to many low-resource languages. Some of these models employ language adapters in their formulation, which helps to improve monolingual performance. This formulation restricts the usability of these models on code-switched speech, where two languages are mixed together in the same utterance. We propose ways to effectively fine-tune such models on code-switched speech, by assimilating information from both language adapters at each language adaptation point in the network.
  arXiv  Detail & Related papers  (2023-10-11T12:15:24Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition [62.997667081978825]
  Code-Switching (CS) is referred to the phenomenon of alternately using words and phrases from different languages. We propose a simple yet effective data augmentation in which audio and corresponding labels of different source languages are transcribed. We show that this augmentation can even improve the model's performance on inter-sentential language switches not seen during training by 5,03% WER.
  arXiv  Detail & Related papers  (2022-10-17T12:15:57Z)
• LAE: Language-Aware Encoder for Monolingual and Multilingual ASR [87.74794847245536]
  A novel language-aware encoder (LAE) architecture is proposed to handle both situations by disentangling language-specific information. Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level.
  arXiv  Detail & Related papers  (2022-06-05T04:03:12Z)
• Code Switched and Code Mixed Speech Recognition for Indic languages [0.0]
  Training multilingual automatic speech recognition (ASR) systems is challenging because acoustic and lexical information is typically language specific. We compare the performance of end to end multilingual speech recognition system to the performance of monolingual models conditioned on language identification (LID) We also propose a similar technique to solve the Code Switched problem and achieve a WER of 21.77 and 28.27 over Hindi-English and Bengali-English respectively.
  arXiv  Detail & Related papers  (2022-03-30T18:09:28Z)
• Reducing language context confusion for end-to-end code-switching automatic speech recognition [50.89821865949395]
  We propose a language-related attention mechanism to reduce multilingual context confusion for the E2E code-switching ASR model. By calculating the respective attention of multiple languages, our method can efficiently transfer language knowledge from rich monolingual data.
  arXiv  Detail & Related papers  (2022-01-28T14:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.