Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR

• URL: http://arxiv.org/abs/2410.13445v1
• Date: Thu, 17 Oct 2024 11:19:44 GMT
• Title: Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR
• Authors: Abhishek Gupta, Amruta Parulekar, Sameep Chattopadhyay, Preethi Jyothi,
• Abstract summary: Multimodal models are able to leverage unlabeled text via text-only adaptation with further parameter-efficient ASR fine-tuning. We show cross-lingual transfer from a high-resource language, achieving up to a relative 17% WER reduction over a baseline in a zero-shot setting.
• Score: 25.566285376879094
• License:
• Abstract: Automatic speech recognition (ASR) for low-resource languages remains a challenge due to the scarcity of labeled training data. Parameter-efficient fine-tuning and text-only adaptation are two popular methods that have been used to address such low-resource settings. In this work, we investigate how these techniques can be effectively combined using a multilingual multimodal model like SeamlessM4T. Multimodal models are able to leverage unlabeled text via text-only adaptation with further parameter-efficient ASR fine-tuning, thus boosting ASR performance. We also show cross-lingual transfer from a high-resource language, achieving up to a relative 17% WER reduction over a baseline in a zero-shot setting without any labeled speech.

Related papers

• Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking [68.77659513993507]
  We present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy. Our results show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively, and word error rates which are 3.3% and 2.0% lower on these benchmarks.
  arXiv  Detail & Related papers  (2024-09-27T03:31:32Z)
• Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition [2.7247388777405597]
  We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets. We fine-tune the Whisper multilingual ASR model on five high-resource languages and one low-resource language.
  arXiv  Detail & Related papers  (2024-09-25T14:09:09Z)
• Meta-Whisper: Speech-Based Meta-ICL for ASR on Low-Resource Languages [51.12146889808824]
  Meta-Whisper is a novel approach to improve automatic speech recognition for low-resource languages. It enhances Whisper's ability to recognize speech in unfamiliar languages without extensive fine-tuning.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• Efficient Compression of Multitask Multilingual Speech Models [0.0]
  DistilWhisper is able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2.
  arXiv  Detail & Related papers  (2024-05-02T03:11:59Z)
• Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts [14.999359332108767]
  We propose DistilWhisper to bridge the performance gap in ASR for under-represented languages. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters.
  arXiv  Detail & Related papers  (2023-11-02T08:37:30Z)
• Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning [28.592569051244375]
  METHODNS simultaneously achieves strong multilingual scalability and low-resource adaptation ability. Our framework achieves a 0.13$sim$2.41 lower character error rate (CER) with 30% smaller inference overhead over state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2023-06-23T16:23:00Z)
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition [50.93943755401025]
  We propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses.
  arXiv  Detail & Related papers  (2023-01-19T02:37:56Z)
• Learning ASR pathways: A sparse multilingual ASR model [31.147484652643282]
  We present ASR pathways, a sparse multilingual ASR model that activates language-specific sub-networks ("pathways") With the overlapping sub-networks, the shared parameters can also enable knowledge transfer for lower-resource languages via joint multilingual training. Our proposed ASR pathways outperform both dense models and a language-agnostically pruned model, and provide better performance on low-resource languages.
  arXiv  Detail & Related papers  (2022-09-13T05:14:08Z)
• ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion [49.617722668505834]
  We show that our approach permits the application of speech synthesis and voice conversion to improve ASR systems using only one target-language speaker during model training. It is possible to obtain promising ASR training results with our data augmentation method using only a single real speaker in a target language.
  arXiv  Detail & Related papers  (2022-03-29T11:55:30Z)
• Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition [62.94773371761236]
  We consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate. We propose a method of dynamic acoustic unit augmentation based on the BPE-dropout technique. Our monolingual Turkish Conformer established a competitive result with 22.2% character error rate (CER) and 38.9% word error rate (WER)
  arXiv  Detail & Related papers  (2021-03-12T10:10:13Z)
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation [63.16500026845157]
  We introduce speech-to-text translation as an auxiliary task to incorporate additional knowledge of the target language. We show that training ST with human translations is not necessary. Even with pseudo-labels from low-resource MT (200K examples), ST-enhanced transfer brings up to 8.9% WER reduction to direct transfer.
  arXiv  Detail & Related papers  (2020-06-09T19:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.