Shifting from endangerment to rebirth in the Artificial Intelligence Age: An Ensemble Machine Learning Approach for Hawrami Text Classification

• URL: http://arxiv.org/abs/2409.16884v1
• Date: Wed, 25 Sep 2024 12:52:21 GMT
• Title: Shifting from endangerment to rebirth in the Artificial Intelligence Age: An Ensemble Machine Learning Approach for Hawrami Text Classification
• Authors: Aram Khaksar, Hossein Hassani,
• Abstract summary: Hawrami, a dialect of Kurdish, is classified as an endangered language. This paper introduces various text classification models using a dataset of 6,854 articles in Hawrami labeled into 15 categories by two native speakers.
• Score: 1.174020933567308
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Hawrami, a dialect of Kurdish, is classified as an endangered language as it suffers from the scarcity of data and the gradual loss of its speakers. Natural Language Processing projects can be used to partially compensate for data availability for endangered languages/dialects through a variety of approaches, such as machine translation, language model building, and corpora development. Similarly, NLP projects such as text classification are in language documentation. Several text classification studies have been conducted for Kurdish, but they were mainly dedicated to two particular dialects: Sorani (Central Kurdish) and Kurmanji (Northern Kurdish). In this paper, we introduce various text classification models using a dataset of 6,854 articles in Hawrami labeled into 15 categories by two native speakers. We use K-nearest Neighbor (KNN), Linear Support Vector Machine (Linear SVM), Logistic Regression (LR), and Decision Tree (DT) to evaluate how well those methods perform the classification task. The results indicate that the Linear SVM achieves a 96% of accuracy and outperforms the other approaches.

Related papers

• Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects [72.18753241750964]
  Yorub'a is an African language with roughly 47 million speakers. Recent efforts to develop NLP technologies for African languages have focused on their standard dialects. We take steps towards bridging this gap by introducing a new high-quality parallel text and speech corpus.
  arXiv  Detail & Related papers  (2024-06-27T22:38:04Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectic datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification [50.675552118811]
  Cross-lingual text classification is typically built on large-scale, multilingual language models (LMs) pretrained on a variety of languages of interest. We propose revisiting the classic "translate-and-test" pipeline to neatly separate the translation and classification stages.
  arXiv  Detail & Related papers  (2023-06-08T07:33:22Z)
• Enhancing Pashto Text Classification using Language Processing Techniques for Single And Multi-Label Analysis [0.0]
  This study aims to establish an automated classification system for Pashto text. The study achieved an average testing accuracy rate of 94%. The use of pre-trained language representation models, such as DistilBERT, showed promising results.
  arXiv  Detail & Related papers  (2023-05-04T23:11:31Z)
• Tuning Traditional Language Processing Approaches for Pashto Text Classification [0.0]
  The main aim of this study is to establish a Pashto automatic text classification system. This study compares several models containing both statistical and neural network machine learning techniques. This research obtained average testing accuracy rate 94% using classification algorithm and TFIDF feature extraction method.
  arXiv  Detail & Related papers  (2023-05-04T22:57:45Z)
• MasakhaNEWS: News Topic Classification for African languages [15.487928928173098]
  African languages are severely under-represented in NLP research due to lack of datasets covering several NLP tasks. We develop MasakhaNEWS -- a new benchmark dataset for news topic classification covering 16 languages widely spoken in Africa.
  arXiv  Detail & Related papers  (2023-04-19T21:12:23Z)
• No Language Left Behind: Scaling Human-Centered Machine Translation [69.28110770760506]
  We create datasets and models aimed at narrowing the performance gap between low and high-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-11T07:33:36Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)
• KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi [18.01565807026177]
  We introduce two news datasets for classification of news articles in Kinyarwanda and Kirundi, two low-resource African languages. We provide statistics, guidelines for preprocessing, and monolingual and cross-lingual baseline models. Our experiments show that training embeddings on the relatively higher-resourced Kinyarwanda yields successful cross-lingual transfer to Kirundi.
  arXiv  Detail & Related papers  (2020-10-23T05:37:42Z)
• Leveraging Multilingual News Websites for Building a Kurdish Parallel Corpus [0.6445605125467573]
  We present a corpus containing 12,327 translation pairs in the two major dialects of Kurdish, Sorani and Kurmanji. We also provide 1,797 and 650 translation pairs in English-Kurmanji and English-Sorani.
  arXiv  Detail & Related papers  (2020-10-04T11:52:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.