Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

• URL: http://arxiv.org/abs/2603.18863v1
• Date: Thu, 19 Mar 2026 13:10:39 GMT
• Title: Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders
• Abstract summary: We show that explicit alignment techniques frequently fail to improve token-level downstream performance.<n>We analyze four XLM-R encoder models aligned on different language pairs and fine-tuned for either POS Tagging or Sentence Classification.
• Score: 49.098124747769276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Better cross-lingual alignment is often assumed to yield better cross-lingual transfer. However, explicit alignment techniques -- despite increasing embedding similarity -- frequently fail to improve token-level downstream performance. In this work, we show that this mismatch arises because alignment and downstream task objectives are largely orthogonal, and because the downstream benefits from alignment vary substantially across languages and task types. We analyze four XLM-R encoder models aligned on different language pairs and fine-tuned for either POS Tagging or Sentence Classification. Using representational analyses, including embedding distances, gradient similarities, and gradient magnitudes for both task and alignment losses, we find that: (1) embedding distances alone are unreliable predictors of improvements (or degradations) in task performance and (2) alignment and task gradients are often close to orthogonal, indicating that optimizing one objective may contribute little to optimizing the other. Taken together, our findings explain why ``better'' alignment often fails to translate into ``better'' cross-lingual transfer. Based on these insights, we provide practical guidelines for combining cross-lingual alignment with task-specific fine-tuning, highlighting the importance of careful loss selection.

Related papers

• How Transliterations Improve Crosslingual Alignment [48.929677368744606]
  Recent studies have shown that post-aligning multilingual pretrained language models (mPLMs) using alignment objectives can improve crosslingual alignment.<n>This paper attempts to explicitly evaluate the crosslingual alignment and identify the key elements in transliteration-based approaches that contribute to better performance.
  arXiv  Detail & Related papers  (2024-09-25T20:05:45Z)
• Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing [68.47787275021567]
  Cross-lingual semantic parsing transfers parsing capability from a high-resource language (e.g., English) to low-resource languages with scarce training data. We propose a new approach to cross-lingual semantic parsing by explicitly minimizing cross-lingual divergence between latent variables using Optimal Transport.
  arXiv  Detail & Related papers  (2023-07-09T04:52:31Z)
• Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers [0.6882042556551609]
  multilingual language models can achieve cross-lingual transfer without explicit cross-lingual training data. One common way to improve this transfer is to perform realignment steps before fine-tuning. But realignment methods were found to not always improve results across languages and tasks.
  arXiv  Detail & Related papers  (2023-06-05T11:35:40Z)
• Dual-Alignment Pre-training for Cross-lingual Sentence Embedding [79.98111074307657]
  We propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding. We introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. Our approach can significantly improve sentence embedding.
  arXiv  Detail & Related papers  (2023-05-16T03:53:30Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings [7.026476782041066]
  We propose using Optimal Transport (OT) as an alignment objective during fine-tuning to improve multilingual contextualized representations. This approach does not require word-alignment pairs prior to fine-tuning and instead learns the word alignments within context in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-06T16:13:45Z)
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning [61.29879000628815]
  We show that it is crucial for tasks to align gradients between them in order to maximize knowledge transfer. We propose a simple yet effective method that can efficiently align gradients between tasks. We extensively validate our method on various multi-task learning and zero-shot cross-lingual transfer tasks.
  arXiv  Detail & Related papers  (2021-10-06T09:10:10Z)
• Do Explicit Alignments Robustly Improve Multilingual Encoders? [22.954688396858085]
  multilingual encoders can effectively learn cross-lingual representation. Explicit alignment objectives based on bitexts like Europarl or MultiUN have been shown to further improve these representations. We propose a new contrastive alignment objective that can better utilize such signal.
  arXiv  Detail & Related papers  (2020-10-06T07:43:17Z)
• Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study [2.101267270902429]
  We analyse how different forms of cross-lingual supervision and various alignment methods influence the transfer capability of mBERT in zero-shot setting. We find that supervision from parallel corpus is generally superior to dictionary alignments.
  arXiv  Detail & Related papers  (2020-09-29T20:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.