CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

• URL: http://arxiv.org/abs/2605.26293v1
• Date: Mon, 25 May 2026 19:30:45 GMT
• Title: CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
• Abstract summary: Controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English.<n>We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks.
• Score: 15.323051310503688
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, both tuned models win against their respective base across 11 evaluated languages. Overall, we show promising directions for multilingual preference tuning.

Related papers

• Language Chain in Alignment: Cross-lingual Ranking Preference Optimization [23.860944668713696]
  We propose a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language.<n>We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences.<n>Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization.
  arXiv  Detail & Related papers  (2026-08-24T11:54:36Z)
• CM-Align: Consistency-based Multilingual Alignment for Large Language Models [84.19366314925593]
  We propose a consistency-based data method to construct high-quality multilingual preference data.<n> Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction.
  arXiv  Detail & Related papers  (2025-09-10T12:40:49Z)
• Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters [53.59868121093848]
  We introduce Seed-X, a family of open-source language models (LLMs) with 7B parameter size.<n>The base model is pre-trained on a diverse, high-quality dataset encompassing both monolingual and bilingual content across 28 languages.<n>The instruct model is then finetuned to translate by Chain-of-Thought (CoT) reasoning and further enhanced through reinforcement learning (RL) to achieve better generalization across diverse language pairs.
  arXiv  Detail & Related papers  (2025-07-18T03:19:43Z)
• Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment [35.1576728251478]
  We propose a novel approach that captures preferences from well-aligned English models by implicit rewards and transfers them to other languages through iterative training.<n>Fine-tuning Llama3 for two iterations resulted in a 12.72% average improvement in Win Rate and a 5.97% increase in Length Control Win Rate across all training languages on the X-AlpacaEval leaderboard.
  arXiv  Detail & Related papers  (2025-03-06T17:33:01Z)
• AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thought [40.16140566668239]
  We introduce AdaMCOT, a framework that enhances multilingual factual reasoning.<n>AdaMCOT dynamically routing thought processes in intermediary "thinking languages" before generating target-language responses.<n>Our evaluation demonstrates substantial improvements in both factual reasoning quality and cross-lingual consistency.
  arXiv  Detail & Related papers  (2025-01-27T15:48:57Z)
• MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization [65.31411639849516]
  We propose a Multilingual-Alignment-as-Preference Optimization framework (MAPO) to align the reasoning processes in other languages with the dominant language. Specifically, we harness an off-the-shelf translation model for the consistency between answers in non-dominant and dominant languages. Experiments show that MAPO stably achieves significant improvements in the multilingual reasoning of various models.
  arXiv  Detail & Related papers  (2024-01-12T18:03:54Z)
• Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed? [40.13166574854085]
  We investigate the minimal amount of multilinguality required to elicit cross-lingual generalisation in English-centric large language models. We find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation.
  arXiv  Detail & Related papers  (2023-12-20T00:49:52Z)
• Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models [95.32691891392903]
  In this paper, we do cross-lingual evaluation on various NLU tasks using prompt-tuning and compare it with fine-tuning. The results show that prompt tuning achieves much better cross-lingual transfer than fine-tuning across datasets.
  arXiv  Detail & Related papers  (2022-10-22T05:48:02Z)
• Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation [129.99918589405675]
  Large multilingual models trained with self-supervision achieve state-of-the-art results in a wide range of natural language processing tasks. Multilingual fine-tuning improves performance on low-resource languages but requires modifying the entire model and can be prohibitively expensive. We propose training language-family adapters on top of mBART-50 to facilitate cross-lingual transfer.
  arXiv  Detail & Related papers  (2022-09-30T05:02:42Z)
• XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning [68.57658225995966]
  Cross-lingual Choice of Plausible Alternatives (XCOPA) is a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods falls short compared to translation-based transfer.
  arXiv  Detail & Related papers  (2020-05-01T12:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.