A Primer on Pretrained Multilingual Language Models

• URL: http://arxiv.org/abs/2107.00676v1
• Date: Thu, 1 Jul 2021 18:01:46 GMT
• Title: A Primer on Pretrained Multilingual Language Models
• Authors: Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra
• Abstract summary: Multilingual Language Models (MLLMs) have emerged as a viable option for bringing the power of pretraining to a large number of languages. We review the existing literature covering the above broad areas of research pertaining to MLLMs.
• Score: 18.943173499882885
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual Language Models (MLLMs) such as mBERT, XLM, XLM-R, \textit{etc.} have emerged as a viable option for bringing the power of pretraining to a large number of languages. Given their success in zero shot transfer learning, there has emerged a large body of work in (i) building bigger MLLMs covering a large number of languages (ii) creating exhaustive benchmarks covering a wider variety of tasks and languages for evaluating MLLMs (iii) analysing the performance of MLLMs on monolingual, zero shot crosslingual and bilingual tasks (iv) understanding the universal language patterns (if any) learnt by MLLMs and (v) augmenting the (often) limited capacity of MLLMs to improve their performance on seen or even unseen languages. In this survey, we review the existing literature covering the above broad areas of research pertaining to MLLMs. Based on our survey, we recommend some promising directions of future research.

Related papers

• Multilingual Large Language Models: A Systematic Survey [38.972546467173565]
  This paper provides a comprehensive survey of the latest research on multilingual large language models (MLLMs) We first discuss the architecture and pre-training objectives of MLLMs, highlighting the key components and methodologies that contribute to their multilingual capabilities. We present a detailed taxonomy and roadmap covering the assessment of MLLMs' cross-lingual knowledge, reasoning, alignment with human values, safety, interpretability and specialized applications.
  arXiv  Detail & Related papers  (2024-11-17T13:21:26Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language. Currently, instruction-tuned large language models (LLMs) excel at various English tasks. Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• A Survey of Large Language Models for European Languages [4.328283741894074]
  Large Language Models (LLMs) have gained significant attention due to their high performance on a wide range of natural language tasks. We present an overview of LLM families, including LLaMA, PaLM, GPT, and MoE. We provide a comprehensive summary of common monolingual and multilingual datasets used for pretraining large language models.
  arXiv  Detail & Related papers  (2024-08-27T13:10:05Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners [67.85635044939836]
  Large Language Models (LLMs) have shown impressive language capabilities. In this work, we investigate the spontaneous multilingual alignment improvement of LLMs. We find that LLMs instruction-tuned on the question translation data (i.e. without annotated answers) are able to encourage the alignment between English and a wide range of languages.
  arXiv  Detail & Related papers  (2024-05-22T16:46:19Z)
• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias [5.104497013562654]
  We present an overview of MLLMs, covering their evolution, key techniques, and multilingual capacities. We explore widely utilized multilingual corpora for MLLMs' training and multilingual datasets oriented for downstream tasks. We discuss bias on MLLMs including its category and evaluation metrics, and summarize the existing debiasing techniques.
  arXiv  Detail & Related papers  (2024-04-01T05:13:56Z)
• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models [117.20416338476856]
  Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. We propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Our findings indicate that LLMs' proficiency in processing a particular language is predominantly due to a small subset of neurons.
  arXiv  Detail & Related papers  (2024-02-26T09:36:05Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages [51.301942056881146]
  We investigate how large language models (LLMs) function as rerankers in cross-lingual information retrieval systems for African languages. Our implementation covers English and four African languages (Hausa, Somali, Swahili, and Yoruba) We examine cross-lingual reranking with queries in English and passages in the African languages.
  arXiv  Detail & Related papers  (2023-12-26T18:38:54Z)
• Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs [16.770697902481107]
  Large Language Models (LLMs) have demonstrated exceptional natural language understanding abilities. We propose a systematic way of qualifying the performance disparities of LLMs under multilingual settings. The results show that GPT exhibits highly translating-like behaviour in multilingual settings.
  arXiv  Detail & Related papers  (2023-05-24T02:05:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.