Language Chain in Alignment: Cross-lingual Ranking Preference Optimization

• URL: http://arxiv.org/abs/2608.23149v2
• Date: Thu, 27 Aug 2026 10:14:20 GMT
• Title: Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
• Abstract summary: We propose a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language.<n>We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences.<n>Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization.
• Score: 23.860944668713696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.

Related papers

• Meta-Learning Preferences for Multilingual LLM Alignment [31.4144889231502]
  Unequal availability of human preference data across languages poses a challenge for aligning large language models in multilingual settings.<n>We propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization.
  arXiv  Detail & Related papers  (2026-07-14T22:38:39Z)
• Optimizing Language Models for Crosslingual Knowledge Consistency [90.86445137816942]
  Large language models are known to often exhibit inconsistent knowledge.<n>This is particularly problematic in multilingual scenarios, where models are likely to be asked similar questions in different languages.<n>In this work, we show that this issue can be mitigated using reinforcement learning with a structured reward function.
  arXiv  Detail & Related papers  (2026-03-04T23:36:55Z)
• CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences [4.460583138505673]
  Preference optimization is used to align large language models with human preferences, typically by fine-tuning on ranked response pairs.<n>We propose Confidence-Aware Preference Optimization (CAPO), which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism.<n>CAPO enhances robustness to noisy or low-margin comparisons, typically encountered in multilingual text.
  arXiv  Detail & Related papers  (2025-11-10T23:28:12Z)
• CM-Align: Consistency-based Multilingual Alignment for Large Language Models [84.19366314925593]
  We propose a consistency-based data method to construct high-quality multilingual preference data.<n> Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction.
  arXiv  Detail & Related papers  (2025-09-10T12:40:49Z)
• CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation [20.959453238159863]
  Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation (MT) remains challenging.<n>Direct Preference Optimization (DPO) has emerged as a simpler and more efficient alternative, but its performance depends heavily on the quality of preference data.<n>We propose Confidence-Reward driven Preference Optimization (CRPO), a novel method that combines reward scores with model confidence to improve data selection for fine-tuning.
  arXiv  Detail & Related papers  (2025-01-23T18:59:47Z)
• Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization [1.1368382184602488]
  The automatic generation of counter-speech (CS) is a critical strategy for addressing hate speech by providing constructive and informed responses.<n>Existing methods often fail to generate high-quality, impactful, and scalable CS, particularly across diverse linguistic contexts.<n>We propose a novel methodology to enhance CS generation by aligning Large Language Models (LLMs) using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO)
  arXiv  Detail & Related papers  (2024-12-19T23:22:11Z)
• Preference Alignment Improves Language Model-Based TTS [76.70693823683091]
  preference alignment algorithms adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores.
  arXiv  Detail & Related papers  (2024-09-19T01:58:19Z)
• Unintended Impacts of LLM Alignment on Global Representation [62.6579934112071]
  We show that developers align Large Language Models (LLMs) to user preferences through a variety of procedures, such as Reinforcement Learning From Human Feedback (RLHF) and Direct Preference Optimization (DPO) We explore how alignment impacts performance along three axes of global representation: English dialects, multilingualism, and opinions from and about countries worldwide. We conclude by discussing design decisions that led to these unintended impacts and recommendations for more equitable preference tuning.
  arXiv  Detail & Related papers  (2024-02-22T23:31:22Z)
• MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization [65.31411639849516]
  We propose a Multilingual-Alignment-as-Preference Optimization framework (MAPO) to align the reasoning processes in other languages with the dominant language. Specifically, we harness an off-the-shelf translation model for the consistency between answers in non-dominant and dominant languages. Experiments show that MAPO stably achieves significant improvements in the multilingual reasoning of various models.
  arXiv  Detail & Related papers  (2024-01-12T18:03:54Z)
• Distributionally Robust Multilingual Machine Translation [94.51866646879337]
  We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization. We show how to practically optimize this objective for large translation corpora using an iterated best response scheme. Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
  arXiv  Detail & Related papers  (2021-09-09T03:48:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.