Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review

• URL: http://arxiv.org/abs/2505.04531v2
• Date: Tue, 08 Jul 2025 14:57:13 GMT
• Title: Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review
• Authors: Josh McGiff, Nikola S. Nikolov,
• Abstract summary: This paper focuses on strategies to address data scarcity in generative language modelling for low-resource languages (LRL)<n>We identify, categorise and evaluate technical approaches, including monolingual data augmentation, back-translation, multilingual training, and prompt engineering.<n>We conclude with recommendations for extending these methods to a wider range of LRLs and outline open challenges in building equitable generative language systems.
• Score: 0.7366405857677227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative language modelling has surged in popularity with the emergence of services such as ChatGPT and Google Gemini. While these models have demonstrated transformative potential in productivity and communication, they overwhelmingly cater to high-resource languages like English. This has amplified concerns over linguistic inequality in natural language processing (NLP). This paper presents the first systematic review focused specifically on strategies to address data scarcity in generative language modelling for low-resource languages (LRL). Drawing from 54 studies, we identify, categorise and evaluate technical approaches, including monolingual data augmentation, back-translation, multilingual training, and prompt engineering, across generative tasks. We also analyse trends in architecture choices, language family representation, and evaluation methods. Our findings highlight a strong reliance on transformer-based models, a concentration on a small subset of LRLs, and a lack of consistent evaluation across studies. We conclude with recommendations for extending these methods to a wider range of LRLs and outline open challenges in building equitable generative language systems. Ultimately, this review aims to support researchers and developers in building inclusive AI tools for underrepresented languages, a necessary step toward empowering LRL speakers and the preservation of linguistic diversity in a world increasingly shaped by large-scale language technologies.

Related papers

• Bridging Gaps in Natural Language Processing for Yorùbá: A Systematic Review of a Decade of Progress and Prospects [0.6554326244334868]
  This review highlights the scarcity of annotated corpora, limited availability of pre-trained language models, and linguistic challenges like tonal complexity and diacritic dependency as significant obstacles.<n>The findings reveal a growing body of multilingual and monolingual resources, even though the field is constrained by socio-cultural factors such as code-switching and desertion of language for digital usage.
  arXiv  Detail & Related papers  (2025-02-24T17:41:48Z)
• LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models [62.47865866398233]
  This white paper proposes a framework to generate linguistic tools for low-resource languages. By addressing the data scarcity that hinders intelligent applications for such languages, we contribute to promoting linguistic diversity.
  arXiv  Detail & Related papers  (2024-11-20T16:59:41Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• LOLA -- An Open-Source Massively Multilingual Large Language Model [1.5704590739448838]
  LOLA is a massively multilingual large language model trained on more than 160 languages.<n>Our architectural and implementation choices address the challenge of harnessing linguistic diversity.<n>We show how the learned expert-routing mechanism exploits implicit phylogenetic patterns to potentially alleviate the curse of multilinguality.
  arXiv  Detail & Related papers  (2024-09-17T15:23:08Z)
• MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting [53.77590764277568]
  We introduce a novel MoE-CT architecture that separates the base model's learning from the multilingual expansion process. Our design freezes the original LLM parameters, thus safeguarding its performance in high-resource languages, while an appended MoE module, trained on diverse language datasets, augments low-resource language proficiency.
  arXiv  Detail & Related papers  (2024-06-25T11:03:45Z)
• Open Generative Large Language Models for Galician [1.3049334790726996]
  Large language models (LLMs) have transformed natural language processing. Yet, their predominantly English-centric training has led to biases and performance disparities across languages. This imbalance marginalizes minoritized languages, making equitable access to NLP technologies more difficult for languages with lower resources, such as Galician. We present the first two generative LLMs focused on Galician to bridge this gap.
  arXiv  Detail & Related papers  (2024-06-19T23:49:56Z)
• A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers [51.8203871494146]
  The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing.<n>Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient.<n>This survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.
  arXiv  Detail & Related papers  (2024-05-17T17:47:39Z)
• DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models [76.79929883963275]
  DIALIGHT is a toolkit for developing and evaluating multilingual Task-Oriented Dialogue (ToD) systems. It features a secure, user-friendly web interface for fine-grained human evaluation at both local utterance level and global dialogue level. Our evaluations reveal that while PLM fine-tuning leads to higher accuracy and coherence, LLM-based systems excel in producing diverse and likeable responses.
  arXiv  Detail & Related papers  (2024-01-04T11:27:48Z)
• A Survey of Large Language Models [81.06947636926638]
  Language modeling has been widely studied for language understanding and generation in the past two decades. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size.
  arXiv  Detail & Related papers  (2023-03-31T17:28:46Z)
• Overcoming Language Disparity in Online Content Classification with Multimodal Learning [22.73281502531998]
  Large language models are now the standard to develop state-of-the-art solutions for text detection and classification tasks. The development of advanced computational techniques and resources is disproportionately focused on the English language. We explore the promise of incorporating the information contained in images via multimodal machine learning.
  arXiv  Detail & Related papers  (2022-05-19T17:56:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.