BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment

• URL: http://arxiv.org/abs/2411.16300v1
• Date: Mon, 25 Nov 2024 11:35:08 GMT
• Title: BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment
• Authors: Shaolei Zhang, Kehao Zhang, Qingkai Fang, Shoutao Guo, Yan Zhou, Xiaodong Liu, Yang Feng,
• Abstract summary: We introduce BayLing 2, which efficiently transfers generative capabilities and knowledge from high-resource languages to low-resource languages. For multilingual translation across 100+ languages, BayLing shows superior performance compared to open-source models of similar scale. Demo, homepage, code and models of BayLing are available.
• Score: 42.193395498828764
• License:
• Abstract: Large language models (LLMs), with their powerful generative capabilities and vast knowledge, empower various tasks in everyday life. However, these abilities are primarily concentrated in high-resource languages, leaving low-resource languages with weaker generative capabilities and relatively limited knowledge. Enhancing the multilingual capabilities of LLMs is therefore crucial for serving over 100 linguistic communities worldwide. An intuitive approach to enhance the multilingual capabilities would be to construct instruction data for various languages, but constructing instruction data for over 100 languages is prohibitively costly. In this paper, we introduce BayLing 2, which efficiently transfers generative capabilities and knowledge from high-resource languages to low-resource languages through language alignment. To achieve this, we constructed a dataset of 3.2 million instructions, comprising high-resource language instructions (Chinese and English) and cross-lingual instructions for 100+ languages and performed instruction tuning based on the dataset to facilitate the capability transfer between languages. Using Llama as the foundation model, we developed BayLing-2-7B, BayLing-2-13B, and BayLing-3-8B, and conducted a comprehensive evaluation of BayLing. For multilingual translation across 100+ languages, BayLing shows superior performance compared to open-source models of similar scale. For multilingual knowledge and understanding benchmarks, BayLing achieves significant improvements across over 20 low-resource languages, demonstrating its capability of effective knowledge transfer from high-resource to low-resource languages. Furthermore, results on English benchmarks indicate that BayLing maintains high performance in highresource languages while enhancing the performance in low-resource languages. Demo, homepage, code and models of BayLing are available.

Related papers

• Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysis [19.37853222555255]
  Probing techniques for large language models (LLMs) have primarily focused on English, overlooking the vast majority of the world's languages. We conduct experiments on several open-source LLM models, analyzing probing accuracy, trends across layers, and similarities between probing vectors for multiple languages.
  arXiv  Detail & Related papers  (2024-09-22T14:14:05Z)
• Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages [60.162717568496355]
  Large language models (LLMs) have been pre-trained on multilingual corpora. Their performance still lags behind in most languages compared to a few resource-rich languages.
  arXiv  Detail & Related papers  (2024-02-19T15:07:32Z)
• PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning [46.153828074152436]
  We propose a pivot language guided generation approach to enhance instruction tuning in lower-resource languages. It trains the model to first process instructions in the pivot language, and then produce responses in the target language. Our approach demonstrates a significant improvement in the instruction-following abilities of LLMs by 29% on average.
  arXiv  Detail & Related papers  (2023-11-15T05:28:07Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• Adapters for Enhanced Modeling of Multilingual Knowledge and Text [54.02078328453149]
  Language models have been extended to multilingual language models (MLLMs) Knowledge graphs contain facts in an explicit triple format, which require careful curation and are only available in a few high-resource languages. We propose to enhance MLLMs with knowledge from multilingual knowledge graphs (MLKGs) so as to tackle language and knowledge graph tasks across many languages.
  arXiv  Detail & Related papers  (2022-10-24T21:33:42Z)
• Prix-LM: Pretraining for Multilingual Knowledge Base Construction [59.02868906044296]
  We propose a unified framework, Prix-LM, for multilingual knowledge construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs. Experiments on standard entity-related tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2021-10-16T02:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.