Building low-resource African language corpora: A case study of Kidawida, Kalenjin and Dholuo

• URL: http://arxiv.org/abs/2501.11003v1
• Date: Sun, 19 Jan 2025 10:17:21 GMT
• Title: Building low-resource African language corpora: A case study of Kidawida, Kalenjin and Dholuo
• Authors: Audrey Mbogho, Quin Awuor, Andrew Kipkebut, Lilian Wanzare, Vivian Oloo,
• Abstract summary: This paper presents a case study on the development of linguistic corpora for three under-resourced Kenyan languages, Kidaw'ida, Kalenjin, and Dholuo. Our project employed a selective crowd-sourcing methodology to collect text and speech data from native speakers of these languages. We made these resources freely accessible via open-research platforms, namely Zenodo for the parallel text corpora and Mozilla Common Voice for the speech datasets.
• Score: 0.815557531820863
• License:
• Abstract: Natural Language Processing is a crucial frontier in artificial intelligence, with broad applications in many areas, including public health, agriculture, education, and commerce. However, due to the lack of substantial linguistic resources, many African languages remain underrepresented in this digital transformation. This paper presents a case study on the development of linguistic corpora for three under-resourced Kenyan languages, Kidaw'ida, Kalenjin, and Dholuo, with the aim of advancing natural language processing and linguistic research in African communities. Our project, which lasted one year, employed a selective crowd-sourcing methodology to collect text and speech data from native speakers of these languages. Data collection involved (1) recording conversations and translation of the resulting text into Kiswahili, thereby creating parallel corpora, and (2) reading and recording written texts to generate speech corpora. We made these resources freely accessible via open-research platforms, namely Zenodo for the parallel text corpora and Mozilla Common Voice for the speech datasets, thus facilitating ongoing contributions and access for developers to train models and develop Natural Language Processing applications. The project demonstrates how grassroots efforts in corpus building can support the inclusion of African languages in artificial intelligence innovations. In addition to filling resource gaps, these corpora are vital in promoting linguistic diversity and empowering local communities by enabling Natural Language Processing applications tailored to their needs. As African countries like Kenya increasingly embrace digital transformation, developing indigenous language resources becomes essential for inclusive growth. We encourage continued collaboration from native speakers and developers to expand and utilize these corpora.

Related papers

• LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models [62.47865866398233]
  This white paper proposes a framework to generate linguistic tools for low-resource languages. By addressing the data scarcity that hinders intelligent applications for such languages, we contribute to promoting linguistic diversity.
  arXiv  Detail & Related papers  (2024-11-20T16:59:41Z)
• Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences [31.62071644137294]
  We discuss the decreasing diversity of languages in the world and how working with Indigenous languages poses unique ethical challenges for AI and NLP. We report encouraging results in the development of high-quality machine learning translators for Indigenous languages. We present prototypes we have built in projects done in 2023 and 2024 with Indigenous communities in Brazil, aimed at facilitating writing.
  arXiv  Detail & Related papers  (2024-07-17T14:46:37Z)
• Enhancing Language Learning through Technology: Introducing a New English-Azerbaijani (Arabic Script) Parallel Corpus [0.9051256541674136]
  This paper introduces a pioneering English-Azerbaijani (Arabic Script) parallel corpus. It is designed to bridge the technological gap in language learning and machine translation for under-resourced languages.
  arXiv  Detail & Related papers  (2024-07-06T21:23:20Z)
• Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects [72.18753241750964]
  Yorub'a is an African language with roughly 47 million speakers. Recent efforts to develop NLP technologies for African languages have focused on their standard dialects. We take steps towards bridging this gap by introducing a new high-quality parallel text and speech corpus.
  arXiv  Detail & Related papers  (2024-06-27T22:38:04Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• NusaCrowd: Open Source Initiative for Indonesian NLP Resources [104.5381571820792]
  NusaCrowd is a collaborative initiative to collect and unify existing resources for Indonesian languages. Our work strives to advance natural language processing (NLP) research for languages that are under-represented despite being widely spoken.
  arXiv  Detail & Related papers  (2022-12-19T17:28:22Z)
• Building African Voices [125.92214914982753]
  This paper focuses on speech synthesis for low-resourced African languages. We create a set of general-purpose instructions on building speech synthesis systems with minimum technological resources. We release the speech data, code, and trained voices for 12 African languages to support researchers and developers.
  arXiv  Detail & Related papers  (2022-07-01T23:28:16Z)
• Including Signed Languages in Natural Language Processing [48.62744923724317]
  Signed languages are the primary means of communication for many deaf and hard of hearing individuals. This position paper calls on the NLP community to include signed languages as a research area with high social and scientific impact.
  arXiv  Detail & Related papers  (2021-05-11T17:37:55Z)
• The first large scale collection of diverse Hausa language datasets [0.0]
  Hausa is considered well-studied and documented language among the sub-Saharan African languages. It is estimated that over 100 million people speak the language. We provide an expansive collection of curated datasets consisting of both formal and informal forms of the language.
  arXiv  Detail & Related papers  (2021-02-13T19:34:20Z)
• Google Crowdsourced Speech Corpora and Related Open-Source Resources for Low-Resource Languages and Dialects: An Overview [43.92114369646489]
  We have released 38 datasets for building text-to-speech and automatic speech recognition applications. The paper describes the methodology used for developing such corpora and presents some of our findings that could benefit under-represented language communities.
  arXiv  Detail & Related papers  (2020-10-14T02:24:04Z)
• Towards Neural Machine Translation for Edoid Languages [2.144787054581292]
  Many Nigerian languages have relinquished their previous prestige and purpose in modern society to English and Nigerian Pidgin. This work explores the feasibility of Neural Machine Translation for the Edoid language family of Southern Nigeria.
  arXiv  Detail & Related papers  (2020-03-24T07:53:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.