TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models

• URL: http://arxiv.org/abs/2604.26553v1
• Date: Wed, 29 Apr 2026 11:39:43 GMT
• Title: TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
• Abstract summary: Large language models (LLMs) demonstrate strong multilingual capabilities, yet often fail to consistently generate responses in the intended language.<n>We introduce Token-Level Policy Optimization (TLPO), a fine-tuning framework designed to mitigate language confusion through localized, token-level updates.
• Score: 6.468917070386353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) demonstrate strong multilingual capabilities, yet often fail to consistently generate responses in the intended language, exhibiting a phenomenon known as language confusion. Prior mitigation approaches based on sequence-level fine-tuning, such as DPO, ORPO, and GRPO, operate at the level of entire responses and can lead to unintended degradation of general model capabilities, motivating the need for more fine-grained alternatives. To address this, we introduce Token-Level Policy Optimization (TLPO), a fine-tuning framework designed to mitigate language confusion through localized, token-level updates. TLPO identifies error-prone positions, explores alternative candidate tokens, and updates the policy using a tailored objective to suppress error-inducing outputs at a granular level. This selective intervention enables effective mitigation of language confusion without compromising the model's general abilities. Experiments on multiple multilingual LLMs across diverse languages demonstrate that TLPO significantly outperforms baselines in improving language consistency while preserving downstream task accuracy.

Related papers

• When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use [2.4319925598898133]
  A common failure occurs when a model selects the correct tool but generates argument values in an inconsistent language.<n>We revisit post-training strategies for mitigating Argument Language Mismatch (ALM)<n>We find that supervised fine-tuning improves argument language consistency and end-to-end function call accuracy.
  arXiv  Detail & Related papers  (2026-08-12T06:55:04Z)
• Multilingual Fine-Tuning via Localized Gradient Conflict Resolution [20.41455787951831]
  We introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally.<n> Empirically, Bucket-Level MOOs mitigate interference by driving Large Language Models to construct distinct language-specific dimensions.<n>Our method significantly improves both seen and unseen multilingual performance over standard fine-tuning paradigms.
  arXiv  Detail & Related papers  (2026-06-04T02:36:30Z)
• Learning to Route Languages for Multilingual Policy Optimization [50.044784120224335]
  Large language models are trained on heterogeneous multilingual corpora.<n>We propose language-routed policy optimization (LRPO), an online policy optimization framework that treats language as a selectable variable.
  arXiv  Detail & Related papers  (2026-05-25T02:28:41Z)
• Optimizing Language Models for Crosslingual Knowledge Consistency [90.86445137816942]
  Large language models are known to often exhibit inconsistent knowledge.<n>This is particularly problematic in multilingual scenarios, where models are likely to be asked similar questions in different languages.<n>In this work, we show that this issue can be mitigated using reinforcement learning with a structured reward function.
  arXiv  Detail & Related papers  (2026-03-04T23:36:55Z)
• Language Drift in Multilingual Retrieval-Augmented Generation: Characterization and Decoding-Time Mitigation [11.110312833458421]
  We study output language drift in multilingual RAG across multiple datasets, languages, and LLM backbones.<n>Our experiments reveal that the drift results from decoder-level collapse, where dominant token distributions and high-frequency English patterns dominate the intended generation language.<n>We propose Soft Constrained Decoding (SCD), a lightweight, training-free decoding strategy that gently steers generation toward the target language.
  arXiv  Detail & Related papers  (2025-11-13T05:36:31Z)
• Controlling Language Confusion in Multilingual LLMs [0.0]
  Large language models often suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages.<n>In this work, we apply ORPO, which adds penalties for unwanted output styles to standard SFT, effectively suppressing language-confused generations.
  arXiv  Detail & Related papers  (2025-05-25T12:15:31Z)
• LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangement [21.236557779562794]
  LangPert is a language-based framework designed to detect and mitigate Task-Level Perturbations (TLP)<n>LangPert integrates a Visual Language Model (VLM) to comprehensively monitor policy's skill execution and environmental TLP.<n>Our experimental results demonstrate that LangPert handles diverse TLP situations more effectively than baseline methods.
  arXiv  Detail & Related papers  (2025-04-14T05:39:15Z)
• Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization [1.1368382184602488]
  The automatic generation of counter-speech (CS) is a critical strategy for addressing hate speech by providing constructive and informed responses.<n>Existing methods often fail to generate high-quality, impactful, and scalable CS, particularly across diverse linguistic contexts.<n>We propose a novel methodology to enhance CS generation by aligning Large Language Models (LLMs) using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO)
  arXiv  Detail & Related papers  (2024-12-19T23:22:11Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Efficient Spoken Language Recognition via Multilabel Classification [53.662747523872305]
  We show that our models obtain competitive results while being orders of magnitude smaller and faster than current state-of-the-art methods. Our multilabel strategy is more robust to unseen non-target languages compared to multiclass classification.
  arXiv  Detail & Related papers  (2023-06-02T23:04:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.