Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech

• URL: http://arxiv.org/abs/2606.19381v1
• Date: Sun, 14 Jun 2026 07:56:36 GMT
• Title: Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech
• Abstract summary: This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity.<n> Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning.
• Score: 37.15980892511798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data augmentation via Text-to-speech (TTS) has been explored, existing CS TTS approaches primarily optimise reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed approach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively.

Related papers

• Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech [50.45710815530982]
  Simultaneous speech translation (SST) generates translations while receiving partial speech input.<n>Recent advances show that large language models (LLMs) can substantially improve SST quality, but at the cost of high computational overhead.<n>We propose a Hierarchical Policy Optimization (HPO) approach that trains post-train models on imperfect SFT data.<n> Experiments on English to Chinese/German/Japanese demonstrate improvements of over +7 COMET score and +1.25 MetricX score at a latency of 1.5 seconds.
  arXiv  Detail & Related papers  (2026-04-22T19:43:51Z)
• Improving Code-Switching Speech Recognition with TTS Data Augmentation [58.34842693152991]
  This paper explores multilingual text-to-speech (TTS) models as an effective data augmentation technique to address this shortage.<n>We fine-tune the multilingual CosyVoice2 TTS model on the SEAME dataset to generate synthetic conversational Chinese-English code-switching speech.
  arXiv  Detail & Related papers  (2026-01-02T10:11:51Z)
• Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages [3.263178944046948]
  Code-switching (CS) presents challenges for ASR due to scarce and costly transcribed data.<n>We propose a phrase-level mixing method to generate synthetic CS data that mimics natural patterns.
  arXiv  Detail & Related papers  (2025-06-17T04:37:16Z)
• A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data [46.73430446242378]
  We propose a self-refining framework that enhances ASR performance with only unlabeled datasets.<n>We demonstrate the effectiveness of the framework on Taiwanese Mandarin speech.
  arXiv  Detail & Related papers  (2025-06-10T17:30:32Z)
• Speech collage: code-switched audio generation by collaging monolingual corpora [50.356820349870986]
  Speech Collage is a method that synthesizes CS data from monolingual corpora by splicing audio segments. We investigate the impact of generated data on speech recognition in two scenarios.
  arXiv  Detail & Related papers  (2023-09-27T14:17:53Z)
• Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognition [64.9317368575585]
  This paper proposes a gated recurrent fusion (GRF) method with joint training framework for robust end-to-end ASR. The GRF algorithm is used to dynamically combine the noisy and enhanced features. The proposed method achieves the relative character error rate (CER) reduction of 10.04% over the conventional joint enhancement and transformer method.
  arXiv  Detail & Related papers  (2020-11-09T08:52:05Z)
• Improving Low Resource Code-switched ASR using Augmented Code-switched TTS [29.30430160611224]
  Building Automatic Speech Recognition systems for code-switched speech has recently gained renewed attention. End-to-end systems require large amounts of labeled speech. We report significant improvements in ASR performance achieving absolute word error rate (WER) reductions of up to 5%.
  arXiv  Detail & Related papers  (2020-10-12T09:15:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.