Controlling Language Confusion in Multilingual LLMs

• URL: http://arxiv.org/abs/2505.19116v2
• Date: Sun, 20 Jul 2025 08:57:14 GMT
• Title: Controlling Language Confusion in Multilingual LLMs
• Authors: Nahyun Lee, Yeongseo Woo, Hyunwoo Ko, Guijin Son,
• Abstract summary: Large language models often suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages.<n>In this work, we apply ORPO, which adds penalties for unwanted output styles to standard SFT, effectively suppressing language-confused generations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models often suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages. This critically degrades the user experience, especially in low-resource settings. We hypothesize that this issue stems from limitations in conventional fine-tuning objectives, such as supervised learning, which optimize the likelihood of correct tokens without explicitly penalizing undesired outputs such as cross-lingual mixing. Analysis of loss trajectories during pretraining further reveals that models fail to distinguish between monolingual and language-mixed texts, highlighting the absence of inherent pressure to avoid such confusion. In this work, we apply ORPO, which adds penalties for unwanted output styles to standard SFT, effectively suppressing language-confused generations. ORPO maintains strong language consistency, even under high decoding temperatures, while preserving general QA performance. Our findings suggest that incorporating appropriate penalty terms can effectively mitigate language confusion in multilingual models, particularly in low-resource scenarios.

Related papers

• Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs [4.881694369042022]
  Smoothie-Qwen is a lightweight, post-hoc method that mitigates language bias without retraining.<n>applied to the Qwen model, our method reduces unintended Chinese output by over 95%.
  arXiv  Detail & Related papers  (2025-07-08T05:30:51Z)
• Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages [27.63253872229416]
  We propose a Federated Prompt Tuning Paradigm for multilingual scenarios.<n>Our approach achieves 6.9% higher accuracy with improved data efficiency.<n>These findings underscore the potential of our approach to promote social equality and champion linguistic diversity.
  arXiv  Detail & Related papers  (2025-07-02T05:23:20Z)
• Language Surgery in Multilingual Large Language Models [32.77326546076424]
  Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages.<n>This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers.<n>We propose Inference-Time Language Control (ITLC) to enable precise cross-lingual language control and mitigate language confusion.
  arXiv  Detail & Related papers  (2025-06-14T11:09:50Z)
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models [49.09746599881631]
  We present the first mechanistic interpretability study of language confusion.<n>We show that confusion points (CPs) are central to this phenomenon.<n>We show that editing a small set of critical neurons, identified via comparative analysis with multilingual-tuned models, substantially mitigates confusion.
  arXiv  Detail & Related papers  (2025-05-22T11:29:17Z)
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition [55.2480439325792]
  Speech Meta In-Context LEarning (SMILE) is an innovative framework that combines meta-learning with speech in-context learning (SICL)<n>We show that SMILE consistently outperforms baseline methods in training-free few-shot multilingual ASR tasks.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• Investigating Language-Specific Calibration For Pruning Multilingual Large Language Models [11.421452042888523]
  We compare different calibration languages for pruning multilingual models across diverse languages, tasks, models, and SotA pruning techniques. Our results offer practical suggestions, for example, calibrating in the target language can efficiently retain the language modeling capability but does not necessarily benefit downstream tasks.
  arXiv  Detail & Related papers  (2024-08-26T16:29:13Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation [79.96416609433724]
  Zero-shot translation (ZST) aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem.
  arXiv  Detail & Related papers  (2023-09-28T17:02:36Z)
• Lifting the Curse of Multilinguality by Pre-training Modular Transformers [72.46919537293068]
  multilingual pre-trained models suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We introduce language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. Our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.
  arXiv  Detail & Related papers  (2022-05-12T17:59:56Z)
• Inducing Language-Agnostic Multilingual Representations [61.97381112847459]
  Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. We examine three approaches for this: (i) re-aligning the vector spaces of target languages to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering.
  arXiv  Detail & Related papers  (2020-08-20T17:58:56Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.