Are you speaking my languages? On spoken language adherence in multimodal LLMs

• URL: http://arxiv.org/abs/2606.17281v1
• Date: Mon, 15 Jun 2026 20:44:42 GMT
• Title: Are you speaking my languages? On spoken language adherence in multimodal LLMs
• Abstract summary: Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use.<n>Models often misidentify the output language, compromising transcription fidelity and downstream application quality.<n>We propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output.
• Score: 1.554397085362411
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: (1) zero-shot prompting for robust guidance under uncertainty, (2) supervised fine-tuning (SFT) to improve prompt adherence, and (3) Chain-of-Thought (CoT) reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.

Related papers

• Latent Mechanisms of Language Control in Multilingual Language Models [2.24159028524593]
  We present a comparative study of three methods that identify language-controlling latents in cross-layer transcoders.<n>We find that all three methods effectively manipulate generation language, with FreqSel achieving the strongest overall performance.<n>A knock-out analysis suggests the methods select non-overlapping but each-functional latent subsets, indicating redundancy rather than a single canonical language direction.
  arXiv  Detail & Related papers  (2026-08-31T20:21:21Z)
• From Script to Semantics: Prompting Strategies for African NLI [1.0705399532413615]
  We present a systematic study of prompting strategies for Natural Language Inference (NLI) in Swahili, Yoruba, and Hausa.<n>We evaluate five prompting strategies Baseline (zero-shot), Script-Aware, Language Specific, Contrastive, and Native-Label Self-Translation (NL-STP)<n>We find a significant difference in performance of class wise across strategies with highly neutral class collapse and high prediction skew in some configurations.
  arXiv  Detail & Related papers  (2026-06-02T08:20:10Z)
• LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance [77.58408743830314]
  Reinforcement learning has proven effective for enhancing multi-step reasoning in large language models.<n>But its benefits have not fully translated to multilingual contexts.<n>We develop a novel framework that leverages language-conditioned hints to guide exploration in non-English reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-21T14:47:52Z)
• Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs [4.881694369042022]
  Smoothie-Qwen is a lightweight, post-hoc method that mitigates language bias without retraining.<n>applied to the Qwen model, our method reduces unintended Chinese output by over 95%.
  arXiv  Detail & Related papers  (2025-07-08T05:30:51Z)
• AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thought [40.16140566668239]
  We introduce AdaMCOT, a framework that enhances multilingual factual reasoning.<n>AdaMCOT dynamically routing thought processes in intermediary "thinking languages" before generating target-language responses.<n>Our evaluation demonstrates substantial improvements in both factual reasoning quality and cross-lingual consistency.
  arXiv  Detail & Related papers  (2025-01-27T15:48:57Z)
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition [55.2480439325792]
  Speech Meta In-Context LEarning (SMILE) is an innovative framework that combines meta-learning with speech in-context learning (SICL)<n>We show that SMILE consistently outperforms baseline methods in training-free few-shot multilingual ASR tasks.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement [59.37775534633868]
  We introduce a novel method called language arithmetic, which enables training-free post-processing. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes.
  arXiv  Detail & Related papers  (2024-04-24T08:52:40Z)
• Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models [54.58989938395976]
  We introduce a decomposed prompting approach for sequence labeling tasks.<n>We test our method on the Universal Dependencies part-of-speech tagging dataset for 38 languages.
  arXiv  Detail & Related papers  (2024-02-28T15:15:39Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Reducing language context confusion for end-to-end code-switching automatic speech recognition [50.89821865949395]
  We propose a language-related attention mechanism to reduce multilingual context confusion for the E2E code-switching ASR model. By calculating the respective attention of multiple languages, our method can efficiently transfer language knowledge from rich monolingual data.
  arXiv  Detail & Related papers  (2022-01-28T14:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.