Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition

• URL: http://arxiv.org/abs/2606.06985v1
• Date: Fri, 05 Jun 2026 07:24:39 GMT
• Title: Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition
• Abstract summary: We propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions.<n> Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates.
• Score: 4.573909205189436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.

Related papers

• Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction [0.8749675983608171]
  cascaded multi-talker ASR (MT-ASR) performance is often capped by speaker leakage during separation.<n>We propose a complementary pruning-based paradigm that robustly identifies and removes leakage artifacts.<n>Results on LibriMix, LibriSpeechMix, and the AMI Meeting corpus show our algorithm consistently reduces cpW ER across diverse overlap conditions.
  arXiv  Detail & Related papers  (2026-08-23T03:21:45Z)
• COALA: Robust Contextualized Speech-augmented Language Modeling for ASR via Contrastive Regularizer and Biasing Score Estimation [16.40789238594682]
  COALA is a framework designed to enhance speech-augmented language models (SLMs) in complex multi-entity scenarios.<n>We show that COALA consistently achieves superior contextual biasing performance across various biasing list scales.
  arXiv  Detail & Related papers  (2026-07-09T05:26:22Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions [52.45054413627452]
  TPI-Train is a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling.<n>TPI-Bench is a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy.
  arXiv  Detail & Related papers  (2026-04-19T10:03:42Z)
• Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback [82.70507055599093]
  We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models.<n>Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness.
  arXiv  Detail & Related papers  (2026-01-27T00:55:14Z)
• CBF-AFA: Chunk-Based Multi-SSL Fusion for Automatic Fluency Assessment [0.22499166814992438]
  Automatic fluency assessment (AFA) remains challenging, particularly in capturing speech rhythm, pauses, and disfluencies in non-native speakers.<n>We introduce a chunk-based approach integrating self-supervised learning (SSL) models selected for their complementary strengths in phonetic, prosodic, and noisy speech modeling.<n>Our approach improves F1-score by 2.8 and Pearson correlation by 6.2 points over single SSL baselines on Speechocean762, with gains of 4.2 F1-score and 4.0 Pearson points on Avalinguo.
  arXiv  Detail & Related papers  (2025-06-25T08:39:22Z)
• AdaCS: Adaptive Normalization for Enhanced Code-Switching ASR [1.8533128809847572]
  Intra-sentential code-switching is a significant challenge for Automatic Speech Recognition systems.<n>AdaCS is a normalization model integrates an adaptive bias attention module into encoder-decoder network.<n> Experiments show that AdaCS outperforms previous state-of-the-art method on Vietnamese CS ASR normalization.
  arXiv  Detail & Related papers  (2025-01-13T07:27:00Z)
• Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding [1.07288078404291]
  We propose a natural language understanding approach based on Automatic Speech Recognition (ASR) We improve a noisy-channel model to handle transcription inconsistencies caused by ASR errors. Experiments on four benchmark datasets show that Contrastive and Consistency Learning (CCL) outperforms existing methods.
  arXiv  Detail & Related papers  (2024-05-23T23:10:23Z)
• Generative error correction for code-switching speech recognition using large language models [49.06203730433107]
  Code-switching (CS) speech refers to the phenomenon of mixing two or more languages within the same sentence. We propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem.
  arXiv  Detail & Related papers  (2023-10-17T14:49:48Z)
• Speech collage: code-switched audio generation by collaging monolingual corpora [50.356820349870986]
  Speech Collage is a method that synthesizes CS data from monolingual corpora by splicing audio segments. We investigate the impact of generated data on speech recognition in two scenarios.
  arXiv  Detail & Related papers  (2023-09-27T14:17:53Z)
• BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR [54.23941663326509]
  Frequent speaker changes can make speaker change prediction difficult. We propose boundary-aware serialized output training (BA-SOT) Compared to original SOT, BA-SOT reduces CER/UD-CER by 5.1%/14.0%.
  arXiv  Detail & Related papers  (2023-05-23T06:08:13Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.