Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter

• URL: http://arxiv.org/abs/2309.09443v2
• Date: Tue, 19 Sep 2023 06:19:35 GMT
• Title: Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter
• Authors: Song Li, Yongbin You, Xuezhi Wang, Ke Ding, Guanglu Wan
• Abstract summary: We propose two simple and parameter-efficient methods to enhance language-configurable and language-agnostic multilingual speech recognition. Our experiments demonstrate significant performance improvements across seven languages using our proposed methods.
• Score: 15.039113587886225
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual intelligent assistants, such as ChatGPT, have recently gained popularity. To further expand the applications of multilingual artificial intelligence assistants and facilitate international communication, it is essential to enhance the performance of multilingual speech recognition, which is a crucial component of speech interaction. In this paper, we propose two simple and parameter-efficient methods: language prompt tuning and frame-level language adapter, to respectively enhance language-configurable and language-agnostic multilingual speech recognition. Additionally, we explore the feasibility of integrating these two approaches using parameter-efficient fine-tuning methods. Our experiments demonstrate significant performance improvements across seven languages using our proposed methods.

Related papers

• Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection [49.27067541740956]
  Speech Emotion Recognition (SER) is a crucial component in developing general-purpose AI agents capable of natural human-computer interaction. Building robust multilingual SER systems remains challenging due to the scarcity of labeled data in languages other than English and Chinese. We propose an approach to enhance SER performance in low SER resource languages by leveraging data from high-resource languages.
  arXiv  Detail & Related papers  (2024-09-17T08:36:45Z)
• Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts [14.999359332108767]
  We propose DistilWhisper to bridge the performance gap in ASR for under-represented languages. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters.
  arXiv  Detail & Related papers  (2023-11-02T08:37:30Z)
• MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition [12.23416994447554]
  We present a multi-lingual speech recognition network named Mixture-of-Language-Expert(MoLE) MoLE analyzes linguistic expression from input speech in arbitrary languages, activating a language-specific expert with a lightweight language tokenizer. Based on the reliability, the activated expert and the language-agnostic expert are aggregated to represent language-conditioned embedding.
  arXiv  Detail & Related papers  (2023-02-27T13:26:17Z)
• LAE: Language-Aware Encoder for Monolingual and Multilingual ASR [87.74794847245536]
  A novel language-aware encoder (LAE) architecture is proposed to handle both situations by disentangling language-specific information. Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level.
  arXiv  Detail & Related papers  (2022-06-05T04:03:12Z)
• Integrating Knowledge in End-to-End Automatic Speech Recognition for Mandarin-English Code-Switching [41.88097793717185]
  Code-Switching (CS) is a common linguistic phenomenon in multilingual communities. This paper presents our investigations on end-to-end speech recognition for Mandarin-English CS speech.
  arXiv  Detail & Related papers  (2021-12-19T17:31:15Z)
• Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity [81.51206991542242]
  Cross-lingual transfer offers a compelling way to help bridge this digital divide. Current cross-lingual algorithms have shown success in text-based tasks and speech-related tasks over some low-resource languages. We propose a language similarity approach that can efficiently identify acoustic cross-lingual transfer pairs across hundreds of languages.
  arXiv  Detail & Related papers  (2021-11-02T01:55:17Z)
• Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification [73.5497360800395]
  We develop an end-to-end system that supports multiple languages. We exploit knowledge from a pre-trained multi-lingual natural language processing model.
  arXiv  Detail & Related papers  (2021-09-28T04:43:11Z)
• Are Multilingual Models Effective in Code-Switching? [57.78477547424949]
  We study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching.
  arXiv  Detail & Related papers  (2021-03-24T16:20:02Z)
• Meta-Transfer Learning for Code-Switched Speech Recognition [72.84247387728999]
  We propose a new learning method, meta-transfer learning, to transfer learn on a code-switched speech recognition system in a low-resource setting. Our model learns to recognize individual languages, and transfer them so as to better recognize mixed-language speech by conditioning the optimization on the code-switching data.
  arXiv  Detail & Related papers  (2020-04-29T14:27:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.