MaLA-500: Massive Language Adaptation of Large Language Models

• URL: http://arxiv.org/abs/2401.13303v2
• Date: Wed, 3 Apr 2024 08:23:06 GMT
• Title: MaLA-500: Massive Language Adaptation of Large Language Models
• Authors: Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, Hinrich Schütze,
• Abstract summary: MaLA-500 is a novel large language model designed to cover an extensive range of 534 languages. Our intrinsic evaluation demonstrates that MaLA-500 is better at predicting the given texts of low-resource languages than existing multilingual LLMs.
• Score: 61.440556436524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have advanced the state of the art in natural language processing. However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages. To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages. To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c. Our intrinsic evaluation demonstrates that MaLA-500 is better at predicting the given texts of low-resource languages than existing multilingual LLMs. Moreover, the extrinsic evaluation of in-context learning shows that MaLA-500 outperforms previous LLMs on SIB200 and Taxi1500 by a significant margin, i.e., 11.68% and 4.82% marco-average accuracy across languages. We release MaLA-500 at https://huggingface.co/MaLA-LM

Related papers

• EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models [50.459861376459656]
  EMMA-500 is a large-scale multilingual language model continue-trained on texts across 546 languages. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity.
  arXiv  Detail & Related papers  (2024-09-26T14:40:45Z)
• LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages [36.52198103816494]
  Large Language Models (LLMs) demonstrate remarkable translation capabilities in high-resource language tasks. But their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. We conduct extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages.
  arXiv  Detail & Related papers  (2024-07-08T14:18:28Z)
• Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the Sámi Language [7.289015788793582]
  This work focuses on increasing technological participation for the S'ami language. We draw the attention of the ML community towards the language modeling problem of Ultra Low Resource (ULR) languages. We have compiled the available S'ami language resources from the web to create a clean dataset for training language models.
  arXiv  Detail & Related papers  (2024-05-09T13:54:22Z)
• Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions [52.95579788485411]
  LINGOLLM is a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. We implement LINGOLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LINGOLLM elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions.
  arXiv  Detail & Related papers  (2024-02-28T03:44:01Z)
• TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models [50.40191599304911]
  We propose TransliCo to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script. We show that Furina outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks.
  arXiv  Detail & Related papers  (2024-01-12T15:12:48Z)
• A Benchmark for Learning to Translate a New Language from One Grammar Book [41.1108119653453]
  MTOB is a benchmark for learning to translate between English and Kalamang. It asks a model to learn a language from a single human-readable book of grammar explanations. We demonstrate that baselines using current LLMs are promising but fall short of human performance.
  arXiv  Detail & Related papers  (2023-09-28T16:32:28Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages [8.298465385153527]
  Glot500-m is a horizontally scaled Large Language Models (LLMs) that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline.
  arXiv  Detail & Related papers  (2023-05-20T12:26:41Z)
• Massively Multilingual Shallow Fusion with Large Language Models [62.76735265311028]
  We train a single multilingual language model (LM) for shallow fusion in multiple languages. Compared to a dense LM of similar computation during inference, GLaM reduces the WER of an English long-tail test set by 4.4% relative. In a multilingual shallow fusion task, GLaM improves 41 out of 50 languages with an average relative WER reduction of 3.85%, and a maximum reduction of 10%.
  arXiv  Detail & Related papers  (2023-02-17T14:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.