Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models

• URL: http://arxiv.org/abs/2506.13044v1
• Date: Mon, 16 Jun 2025 02:21:15 GMT
• Title: Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models
• Authors: Muhammad Reza Qorib, Junyi Li, Hwee Tou Ng,
• Abstract summary: Large language models (LLMs) have demonstrated impressive translation capabilities even without being explicitly trained on parallel data.<n>Recent work suggests that it is actually caused by incidental bilingual signals present in the training data.<n>Various methods have been proposed to maximize the utility of parallel data to enhance the multilingual capabilities of multilingual encoder-based and encoder-decoder language models.
• Score: 59.21082876068122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have demonstrated impressive translation capabilities even without being explicitly trained on parallel data. This remarkable property has led some to believe that parallel data is no longer necessary for building multilingual language models. While some attribute this to the emergent abilities of LLMs due to scale, recent work suggests that it is actually caused by incidental bilingual signals present in the training data. Various methods have been proposed to maximize the utility of parallel data to enhance the multilingual capabilities of multilingual encoder-based and encoder-decoder language models. However, some decoder-based LLMs opt to ignore parallel data instead. In this work, we conduct a systematic study on the impact of adding parallel data on LLMs' multilingual capabilities, focusing specifically on translation and multilingual common-sense reasoning. Through controlled experiments, we demonstrate that parallel data can significantly improve LLMs' multilingual capabilities.

Related papers

• From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora [85.44082712798553]
  We introduce a large-scale, high-quality multi-way parallel corpus, TED2025, based on TED Talks.<n>This dataset spans 113 languages, with up to 50 languages aligned in parallel, ensuring extensive multilingual coverage.<n>Experiments show that models trained on multiway parallel data consistently outperform those trained on unaligned multilingual data.
  arXiv  Detail & Related papers  (2025-05-20T07:43:45Z)
• Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders [0.0]
  Most pre-trained Vision-Language (VL) models and training data for the downstream tasks are only available in English.<n>We study the alternative approach: transferring an already trained encoder using parallel data.<n>Our results show that even machine-translated task data are the best on average, caption-like authentic parallel data outperformed it in some languages.
  arXiv  Detail & Related papers  (2025-04-30T14:19:15Z)
• LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy [33.85811169010525]
  Large language models (LLMs) exhibit suboptimal performance on low-resource languages.<n>Recent approaches have leveraged multilingual encoders alongside LLMs by introducing trainable parameters connecting the two models.<n>We propose aname, a framework that integrates representations from all encoder layers.
  arXiv  Detail & Related papers  (2025-02-17T03:45:03Z)
• MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment [48.03702722532143]
  MEXA is a method for assessing the multilingual capabilities of English-centric large language models.<n>MEXA computes the alignment between English and non-English languages using parallel sentences.<n>This alignment can be used to estimate model performance in different languages.
  arXiv  Detail & Related papers  (2024-10-08T09:59:23Z)
• Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation [62.202893186343935]
  We explore what it would take to adapt Large Language Models for low-resource languages. We show that parallel data is critical during both pre-training andSupervised Fine-Tuning (SFT) Our experiments with three LLMs across two low-resourced language groups reveal consistent trends, underscoring the generalizability of our findings.
  arXiv  Detail & Related papers  (2024-08-23T00:59:38Z)
• Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners [67.85635044939836]
  Large Language Models (LLMs) have shown impressive language capabilities. In this work, we investigate the spontaneous multilingual alignment improvement of LLMs. We find that LLMs instruction-tuned on the question translation data (i.e. without annotated answers) are able to encourage the alignment between English and a wide range of languages.
  arXiv  Detail & Related papers  (2024-05-22T16:46:19Z)
• UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset [69.33424532827608]
  Open-source large language models (LLMs) have gained significant strength across diverse fields. In this work, we construct an open-source multilingual supervised fine-tuning dataset. The resulting UltraLink dataset comprises approximately 1 million samples across five languages.
  arXiv  Detail & Related papers  (2024-02-07T05:05:53Z)
• How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning [14.02101305717738]
  Multilingual large language models (MLLMs) are jointly trained on data from many different languages. It remains unclear to what extent, and under which conditions, languages rely on each other's data. We find that MLLMs rely on data from multiple languages from the early stages of fine-tuning and that this reliance gradually increases as fine-tuning progresses.
  arXiv  Detail & Related papers  (2023-05-22T17:47:41Z)
• Language Agnostic Multilingual Information Retrieval with Contrastive Learning [59.26316111760971]
  We present an effective method to train multilingual information retrieval systems. We leverage parallel and non-parallel corpora to improve the pretrained multilingual language models. Our model can work well even with a small number of parallel sentences.
  arXiv  Detail & Related papers  (2022-10-12T23:53:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.