On the limited utility of parallel data for learning shared multilingual representations

• URL: http://arxiv.org/abs/2603.29026v1
• Date: Mon, 30 Mar 2026 21:37:34 GMT
• Title: On the limited utility of parallel data for learning shared multilingual representations
• Abstract summary: parallel data seem to have only a minimal effect on the cross-lingual alignment.<n>Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.
• Score: 4.935445658043885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.

Related papers

• Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment [6.718469075779034]
  We show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus can substantially improve representations for NLU tasks.<n>We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages.
  arXiv  Detail & Related papers  (2026-02-25T03:58:24Z)
• The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining [29.376308590290297]
  We compare the standard web corpus with a monolingual-only version that removes all multilingual documents.<n>We categorize bilingual data into parallel (14%), code-switching (72%), and miscellaneous documents (14%) based on semantic relevance.<n>Our experiments reveal that parallel data almost fully restores translation performance, whereas code-switching contributes minimally.
  arXiv  Detail & Related papers  (2026-01-01T14:52:06Z)
• Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models [59.21082876068122]
  Large language models (LLMs) have demonstrated impressive translation capabilities even without being explicitly trained on parallel data.<n>Recent work suggests that it is actually caused by incidental bilingual signals present in the training data.<n>Various methods have been proposed to maximize the utility of parallel data to enhance the multilingual capabilities of multilingual encoder-based and encoder-decoder language models.
  arXiv  Detail & Related papers  (2025-06-16T02:21:15Z)
• From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora [80.0547333327488]
  We introduce a large-scale, high-quality multi-way parallel corpus, TED2025, based on TED Talks.<n>This dataset spans 113 languages, with up to 50 languages aligned in parallel, ensuring extensive multilingual coverage.<n>Experiments show that models trained on multiway parallel data consistently outperform those trained on unaligned multilingual data.
  arXiv  Detail & Related papers  (2025-05-20T07:43:45Z)
• Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders [0.0]
  Most pre-trained Vision-Language (VL) models and training data for the downstream tasks are only available in English.<n>We study the alternative approach: transferring an already trained encoder using parallel data.<n>Our results show that even machine-translated task data are the best on average, caption-like authentic parallel data outperformed it in some languages.
  arXiv  Detail & Related papers  (2025-04-30T14:19:15Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• Language Agnostic Multilingual Information Retrieval with Contrastive Learning [59.26316111760971]
  We present an effective method to train multilingual information retrieval systems. We leverage parallel and non-parallel corpora to improve the pretrained multilingual language models. Our model can work well even with a small number of parallel sentences.
  arXiv  Detail & Related papers  (2022-10-12T23:53:50Z)
• Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking [84.50302759362698]
  We enhance the transfer learning process by intermediate fine-tuning of pretrained multilingual models. We use parallel and conversational movie subtitles datasets to design cross-lingual intermediate tasks. We achieve impressive improvements (> 20% on goal accuracy) on the parallel MultiWoZ dataset and Multilingual WoZ dataset.
  arXiv  Detail & Related papers  (2021-09-28T11:22:38Z)
• On the Language Neutrality of Pre-trained Multilingual Representations [70.93503607755055]
  We investigate the language-neutrality of multilingual contextual embeddings directly and with respect to lexical semantics. Our results show that contextual embeddings are more language-neutral and, in general, more informative than aligned static word-type embeddings. We show how to reach state-of-the-art accuracy on language identification and match the performance of statistical methods for word alignment of parallel sentences.
  arXiv  Detail & Related papers  (2020-04-09T19:50:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.