Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs

• URL: http://arxiv.org/abs/2605.23975v1
• Date: Wed, 13 May 2026 16:15:33 GMT
• Title: Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs
• Abstract summary: Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities.<n>We apply Direct Preference Optimization to align models, constructing preference pairs in which chosen responses preserve mixed-language content.<n>This alignment yields MER reductions up to 89.6% (in-distribution) and 20.0% (out-of-distribution)
• Score: 16.121226347387246
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities. Focusing on English-Mandarin, we identify three failure modes: language omission, translation-instead-of-transcription, and hallucination. We apply Direct Preference Optimization (DPO) to align models, constructing preference pairs in which chosen responses preserve mixed-language content while rejected responses mimic failure patterns. Training three Audio LLMs on 100K pairs (570 hours), we observe consistent behavioral shifts: models learn to preserve language composition rather than translating when prompted for transcription. This alignment yields MER reductions up to 89.6% (in-distribution) and 20.0% (out-of-distribution). Our findings suggest DPO can effectively elicit correct code-switching transcription behavior from multilingual Audio LLMs.

Related papers

• Are you speaking my languages? On spoken language adherence in multimodal LLMs [1.554397085362411]
  Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use.<n>Models often misidentify the output language, compromising transcription fidelity and downstream application quality.<n>We propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output.
  arXiv  Detail & Related papers  (2026-06-15T20:44:42Z)
• Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox [4.088161686930475]
  We introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks.<n>We show consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied answers.<n>We propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt.
  arXiv  Detail & Related papers  (2026-05-26T23:44:23Z)
• Few-Shot Contrastive Adaptation for Audio Abuse Detection in Low-Resource Indic Languages [3.5238606794194816]
  Abusive speech detection is becoming increasingly important as social media shifts towards voice-based interaction.<n>We investigate whether Contrastive Language-Audio Pre-training can support abusive speech detection directly from audio.
  arXiv  Detail & Related papers  (2026-04-10T08:23:03Z)
• Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease [72.0406069194794]
  Speech representations often encode language-dependent structure that can confound dysarthria detection.<n>We propose a representation-level language shift that aligns source-language self-supervised speech representations with the target-language distribution.<n>We evaluate the approach on oral DDK recordings from Parkinson's disease speech datasets in Czech, German, and Spanish.
  arXiv  Detail & Related papers  (2026-03-23T17:23:39Z)
• PRiSM: Benchmarking Phone Realization in Speech Models [70.82595415252682]
  Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis.<n>We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception.
  arXiv  Detail & Related papers  (2026-01-20T15:00:36Z)
• Languages are Modalities: Cross-Lingual Alignment via Encoder Injection [0.8461674097042394]
  We present a compute efficient language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder.<n>LLINK substantially improves bilingual retrieval and achieves 81.3% preference over the base model.<n>We find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment.
  arXiv  Detail & Related papers  (2025-10-31T07:43:21Z)
• Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillation [50.93756215410832]
  This paper introduces the Language Confusion Gate (LCG), a lightweight, plug-in solution that filters tokens during decoding.<n>The LCG is trained using norm-adjusted self-distillation to predict appropriate language families and apply masking only when needed.
  arXiv  Detail & Related papers  (2025-10-20T14:02:37Z)
• Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs [4.881694369042022]
  Smoothie-Qwen is a lightweight, post-hoc method that mitigates language bias without retraining.<n>applied to the Qwen model, our method reduces unintended Chinese output by over 95%.
  arXiv  Detail & Related papers  (2025-07-08T05:30:51Z)
• Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching [65.74653592668743]
  Finetuning self-supervised multilingual representations reduces absolute word error rates by up to 20%. In circumstances with limited training data finetuning self-supervised representations is a better performing and viable solution.
  arXiv  Detail & Related papers  (2023-11-25T17:05:21Z)
• Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding [53.84948040596055]
  We introduce two related methods to mitigate failure cases with a modified decoding objective. Experiments on the massively multilingual models M2M-100 (418M) and SMaLL-100 show that these methods suppress hallucinations and off-target translations.
  arXiv  Detail & Related papers  (2023-09-13T17:15:27Z)
• Exposing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders [85.80950708769923]
  We probe multilingual language models for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a novel method to expose this knowledge by additionally fine-tuning multilingual models. We report substantial gains on standard benchmarks.
  arXiv  Detail & Related papers  (2022-04-30T13:23:16Z)
• Using heterogeneity in semi-supervised transcription hypotheses to improve code-switched speech recognition [6.224255518500385]
  We show that monolingual data may be more closely matched to one of the languages in the code-switch pair. We propose a semi-supervised approach for code-switched ASR.
  arXiv  Detail & Related papers  (2021-06-14T18:39:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.