CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment

• URL: http://arxiv.org/abs/2503.23777v1
• Date: Mon, 31 Mar 2025 06:52:56 GMT
• Title: CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment
• Abstract summary: We propose a scalable and effective filtering method that selects high-quality preference samples with minimal gradient conflicts across languages.<n>Our method leverages gradient surgery to retain samples aligned with an aggregated multilingual update direction.<n>We integrate CONGRAD into self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages.
• Score: 40.44120745699753
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Naive joint training of large language models (LLMs) for multilingual preference alignment can suffer from negative interference. This is a known issue in multilingual training, where conflicting objectives degrade overall performance. However, the impact of this phenomenon in the context of multilingual preference alignment remains largely underexplored. To address this issue, we propose CONGRAD, a scalable and effective filtering method that selects high-quality preference samples with minimal gradient conflicts across languages. Our method leverages gradient surgery to retain samples aligned with an aggregated multilingual update direction. Additionally, we incorporate a sublinear gradient compression strategy that reduces memory overhead during gradient accumulation. We integrate CONGRAD into self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages. Results show that CONGRAD consistently outperforms strong baselines in both seen and unseen languages, with minimal alignment tax.

Related papers

• Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR [6.818862582399845]
  Fine-tuning on low-resource languages often causes catastrophic forgetting.<n>We propose Unified Gradient Projection, which constrains parameter updates using reference gradients from language-balanced replay.<n>On Whisper-large-v3, it achieves near-zero average forgetting.
  arXiv  Detail & Related papers  (2026-07-13T06:57:02Z)
• Multilingual Fine-Tuning via Localized Gradient Conflict Resolution [20.41455787951831]
  We introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally.<n> Empirically, Bucket-Level MOOs mitigate interference by driving Large Language Models to construct distinct language-specific dimensions.<n>Our method significantly improves both seen and unseen multilingual performance over standard fine-tuning paradigms.
  arXiv  Detail & Related papers  (2026-06-04T02:36:30Z)
• One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging [30.967619128236777]
  We study weight-space merging for multilingual machine translation by fully fine-tuning language model.<n>Our experiments reveal that merging degrades performance, especially when target languages differ.<n>These findings suggest that multilingual fine-tuning may reshape geometry in ways that reduce compatibility with standard weight-space merging assumptions.
  arXiv  Detail & Related papers  (2026-04-03T08:45:26Z)
• Objective Soups: Multilingual Multi-Task Modeling for Speech Processing [69.52720282028385]
  Training a single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks.<n>This paper investigates three multi-objective MSP formulations, which we refer to as textbfobjective soup recipes.<n>Our work demonstrates that hierarchical MOO is a more effective and scalable approach for building state-of-the-art MSP models.
  arXiv  Detail & Related papers  (2025-08-12T07:01:09Z)
• Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model [38.0723521889505]
  Experimental results on 18 to 128 languages show that our method reduces the negative transfer between languages and significantly boosts multilingual performance with fewer parameters.<n>Such language group specialization on experts benefits the new language adaptation and reduces the inference on the previous multilingual knowledge learned.
  arXiv  Detail & Related papers  (2025-06-14T07:56:18Z)
• MR. Guard: Multilingual Reasoning Guardrail using Curriculum Learning [56.79292318645454]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking. This vulnerability is exacerbated in multilingual setting, where multilingual safety-aligned data are often limited. We propose an approach to build a multilingual guardrail with reasoning.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization [81.83460411131931]
  In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models' utility, efficiency, and cost. We propose multilingual adaptive gradient-based tokenization to reduce over-segmentation via adaptive gradient-based subword tokenization.
  arXiv  Detail & Related papers  (2024-07-11T18:59:21Z)
• No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement [59.37775534633868]
  We introduce a novel method called language arithmetic, which enables training-free post-processing. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes.
  arXiv  Detail & Related papers  (2024-04-24T08:52:40Z)
• A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning [6.329304732560936]
  Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries. We propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embeddings without semantic loss.
  arXiv  Detail & Related papers  (2022-10-18T15:36:53Z)
• Multi-Level Contrastive Learning for Cross-Lingual Alignment [35.33431650608965]
  Cross-language pre-trained models such as multilingual BERT (mBERT) have achieved significant performance in various cross-lingual downstream NLP tasks. This paper proposes a multi-level contrastive learning framework to further improve the cross-lingual ability of pre-trained models.
  arXiv  Detail & Related papers  (2022-02-26T07:14:20Z)
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning [61.29879000628815]
  We show that it is crucial for tasks to align gradients between them in order to maximize knowledge transfer. We propose a simple yet effective method that can efficiently align gradients between tasks. We extensively validate our method on various multi-task learning and zero-shot cross-lingual transfer tasks.
  arXiv  Detail & Related papers  (2021-10-06T09:10:10Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• Demystify Optimization Challenges in Multilingual Transformers [21.245418118851884]
  We study optimization challenges from loss landscape and parameter plasticity perspectives. We find that imbalanced training data poses task interference between high and low resource languages. We propose Curvature Aware Task Scaling (CATS) which improves both optimization and generalization especially for low resource.
  arXiv  Detail & Related papers  (2021-04-15T17:51:03Z)
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment [59.995385574274785]
  We show that, contrary to previous belief, negative interference also impacts low-resource languages. We present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference.
  arXiv  Detail & Related papers  (2020-10-06T20:48:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.