Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages

• URL: http://arxiv.org/abs/2506.14177v1
• Date: Tue, 17 Jun 2025 04:37:16 GMT
• Title: Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages
• Authors: Tuan Nguyen, Huy-Dat Tran,
• Abstract summary: Code-switching (CS) presents challenges for ASR due to scarce and costly transcribed data.<n>We propose a phrase-level mixing method to generate synthetic CS data that mimics natural patterns.
• Score: 3.263178944046948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code-switching (CS), common in multilingual settings, presents challenges for ASR due to scarce and costly transcribed data caused by linguistic complexity. This study investigates building CS-ASR using synthetic CS data. We propose a phrase-level mixing method to generate synthetic CS data that mimics natural patterns. Utilizing monolingual augmented with synthetic phrase-mixed CS data to fine-tune large pretrained ASR models (Whisper, MMS, SeamlessM4T). This paper focuses on three under-resourced Southeast Asian language pairs: Malay-English (BM-EN), Mandarin-Malay (ZH-BM), and Tamil-English (TA-EN), establishing a new comprehensive benchmark for CS-ASR to evaluate the performance of leading ASR models. Experimental results show that the proposed training strategy enhances ASR performance on monolingual and CS tests, with BM-EN showing highest gains, then TA-EN and ZH-BM. This finding offers a cost-effective approach for CS-ASR development, benefiting research and industry.

Related papers

• Optimizing ASR for Catalan-Spanish Code-Switching: A Comparative Analysis of Methodologies [9.224033819309708]
  Code-switching (CS), the alternating use of two or more languages, challenges automatic speech recognition (ASR) due to scarce training data and linguistic similarities.<n>We improve ASR for Catalan-Spanish CS by exploring three strategies: (1) generating synthetic CS data, (2) concatenating monolingual audio, and (3) leveraging real CS data with language tokens.<n>Results show that combining a modest amount of synthetic CS data with the dominant language token yields the best transcription performance.
  arXiv  Detail & Related papers  (2025-07-18T12:54:41Z)
• KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization [57.08591486199925]
  This paper presents KIT's submissions to the IWSLT 2025 low-resource track.<n>We develop both cascaded systems, and end-to-end (E2E) Speech Translation systems.<n>Building upon pre-trained models, we fine-tune our systems with different strategies to utilize resources efficiently.
  arXiv  Detail & Related papers  (2025-05-26T08:38:02Z)
• Conditioning LLMs to Generate Code-Switched Text [21.240439045909724]
  Code-switching (CS) is still a critical challenge in Natural Language Processing (NLP)<n>This paper presents a novel methodology to generate CS data using Large Language Models (LLMs)<n>We propose back-translating natural CS sentences into monolingual English, and using the resulting parallel corpus to fine-tune LLMs to turn monolingual sentences into CS.
  arXiv  Detail & Related papers  (2025-02-18T15:04:13Z)
• Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling [50.62091603179394]
  Whisper, one of the most advanced ASR models, handles 99 languages effectively.<n>However, Whisper struggles with unseen languages, those not included in its pre-training.<n>We propose methods that exploit these relationships to enhance ASR performance on unseen languages.
  arXiv  Detail & Related papers  (2024-12-21T04:05:43Z)
• ConCSE: Unified Contrastive Learning and Augmentation for Code-Switched Embeddings [4.68732641979009]
  This paper examines the Code-Switching (CS) phenomenon where two languages intertwine within a single utterance.<n>We highlight that the current Equivalence Constraint (EC) theory for CS in other languages may only partially capture English-Korean CS complexities.<n>We introduce a novel Koglish dataset tailored for English-Korean CS scenarios to mitigate such challenges.
  arXiv  Detail & Related papers  (2024-08-28T11:27:21Z)
• Generative error correction for code-switching speech recognition using large language models [49.06203730433107]
  Code-switching (CS) speech refers to the phenomenon of mixing two or more languages within the same sentence. We propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem.
  arXiv  Detail & Related papers  (2023-10-17T14:49:48Z)
• Speech collage: code-switched audio generation by collaging monolingual corpora [50.356820349870986]
  Speech Collage is a method that synthesizes CS data from monolingual corpora by splicing audio segments. We investigate the impact of generated data on speech recognition in two scenarios.
  arXiv  Detail & Related papers  (2023-09-27T14:17:53Z)
• Code-Switching Text Augmentation for Multilingual Speech Processing [36.302629721413155]
  Code-switching in spoken content has enforced ASR systems to handle mixed input. Recent ASR studies showed the predominance of E2E-ASR using multilingual data to handle CS phenomena. We propose a methodology to augment the monolingual data for artificially generating spoken CS text to improve different speech modules.
  arXiv  Detail & Related papers  (2022-01-07T17:14:19Z)
• Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition [71.96870151495536]
  We propose a novel adversarial reprogramming (AR) approach for low-resource spoken command recognition (SCR) The AR procedure aims to modify the acoustic signals (from the target domain) to repurpose a pretrained SCR model. We evaluate the proposed AR-SCR system on three low-resource SCR datasets, including Arabic, Lithuanian, and dysarthric Mandarin speech.
  arXiv  Detail & Related papers  (2021-10-08T05:07:35Z)
• The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results [9.089285414356969]
  This paper describes the design and main outcomes of the ASRU 2019 Mandarin-English code-switching speech recognition challenge. 500 hours Mandarin speech data and 240 hours Mandarin-English intra-sentencial CS data are released to the participants.
  arXiv  Detail & Related papers  (2020-07-12T05:38:57Z)
• Style Variation as a Vantage Point for Code-Switching [54.34370423151014]
  Code-Switching (CS) is a common phenomenon observed in several bilingual and multilingual communities. We present a novel vantage point of CS to be style variations between both the participating languages. We propose a two-stage generative adversarial training approach where the first stage generates competitive negative examples for CS and the second stage generates more realistic CS sentences.
  arXiv  Detail & Related papers  (2020-05-01T15:53:16Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.