Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

• URL: http://arxiv.org/abs/2602.21543v1
• Date: Wed, 25 Feb 2026 03:58:24 GMT
• Title: Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment
• Authors: Barah Fazili, Koustava Goswami,
• Abstract summary: We show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus can substantially improve representations for NLU tasks.<n>We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages.
• Score: 6.718469075779034
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.

Related papers

• Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters [53.59868121093848]
  We introduce Seed-X, a family of open-source language models (LLMs) with 7B parameter size.<n>The base model is pre-trained on a diverse, high-quality dataset encompassing both monolingual and bilingual content across 28 languages.<n>The instruct model is then finetuned to translate by Chain-of-Thought (CoT) reasoning and further enhanced through reinforcement learning (RL) to achieve better generalization across diverse language pairs.
  arXiv  Detail & Related papers  (2025-07-18T03:19:43Z)
• From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora [80.0547333327488]
  We introduce a large-scale, high-quality multi-way parallel corpus, TED2025, based on TED Talks.<n>This dataset spans 113 languages, with up to 50 languages aligned in parallel, ensuring extensive multilingual coverage.<n>Experiments show that models trained on multiway parallel data consistently outperform those trained on unaligned multilingual data.
  arXiv  Detail & Related papers  (2025-05-20T07:43:45Z)
• Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? [42.37657013017192]
  We show that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. We also conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.
  arXiv  Detail & Related papers  (2024-02-21T11:07:07Z)
• RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training [84.23022072347821]
  We propose a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs. Experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2023-05-13T14:41:05Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• Multilingual Multimodal Learning with Machine Translated Text [27.7207234512674]
  We investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning.
  arXiv  Detail & Related papers  (2022-10-24T11:41:20Z)
• Language Agnostic Multilingual Information Retrieval with Contrastive Learning [59.26316111760971]
  We present an effective method to train multilingual information retrieval systems. We leverage parallel and non-parallel corpora to improve the pretrained multilingual language models. Our model can work well even with a small number of parallel sentences.
  arXiv  Detail & Related papers  (2022-10-12T23:53:50Z)
• Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking [84.50302759362698]
  We enhance the transfer learning process by intermediate fine-tuning of pretrained multilingual models. We use parallel and conversational movie subtitles datasets to design cross-lingual intermediate tasks. We achieve impressive improvements (> 20% on goal accuracy) on the parallel MultiWoZ dataset and Multilingual WoZ dataset.
  arXiv  Detail & Related papers  (2021-09-28T11:22:38Z)
• Improving Low-resource Reading Comprehension via Cross-lingual Transposition Rethinking [0.9236074230806579]
  Extractive Reading (ERC) has made tremendous advances enabled by the availability of large-scale high-quality ERC training data. Despite of such rapid progress and widespread application, the datasets in languages other than high-resource languages such as English remain scarce. We propose a Cross-Lingual Transposition ReThinking (XLTT) model by modelling existing high-quality extractive reading comprehension datasets in a multilingual environment.
  arXiv  Detail & Related papers  (2021-07-11T09:35:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.