Building Arabic NLP from the Ground Up: Twenty Years of Lessons, Failures, and Open Problems

• URL: http://arxiv.org/abs/2605.20786v1
• Date: Wed, 20 May 2026 06:30:16 GMT
• Title: Building Arabic NLP from the Ground Up: Twenty Years of Lessons, Failures, and Open Problems
• Abstract summary: This paper reflects on twenty years of building NLP resources and research infrastructure for Arabic.<n>We discuss three failures: a depression detection corpus that never reached clinical practice, a period of spreading across too many shared tasks without sufficient depth, and a long-standing assumption that Modern Standard Arabic infrastructure would transfer cleanly to dialectal tasks.
• Score: 0.7734073750595959
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper reflects on twenty years of building NLP resources and research infrastructure for Arabic, a language spoken by hundreds of millions yet historically underserved relative to languages such as English or Chinese. The first decade focused on foundational linguistic infrastructure; the second shifted toward computational social science, social media analysis, and socially oriented applications. Rather than cataloguing outputs, the paper examines what the experience of building them revealed. Three counterintuitive lessons emerge: building datasets is as much a social process as a technical one; communities formed around shared tasks often matter more than the tasks themselves; and moving from language resources to computational social science exposes challenges that traditional NLP training does not address. We discuss three failures: a depression detection corpus that never reached clinical practice, a period of spreading across too many shared tasks without sufficient depth, and a long-standing assumption that Modern Standard Arabic infrastructure would transfer cleanly to dialectal tasks. These experiences suggest that the hardest problems in developing NLP for underserved communities are not linguistic but social, institutional, and epistemic, and require competencies the field rarely teaches.

Related papers

• NaijaNLP: A Survey of Nigerian Low-Resource Languages [0.0]
  Three languages -- Hausa, Yorub'a and Igbo -- account for about 60% of the spoken languages in Nigeria.<n>These languages are categorised as low-resource due to insufficient resources to support tasks in computational linguistics.<n>This study presents the first comprehensive review of advancements in low-resource NLP (LR-NLP) research across the three major Nigerian languages.
  arXiv  Detail & Related papers  (2025-02-27T05:48:51Z)
• Bridging Gaps in Natural Language Processing for Yorùbá: A Systematic Review of a Decade of Progress and Prospects [0.6554326244334868]
  This review highlights the scarcity of annotated corpora, limited availability of pre-trained language models, and linguistic challenges like tonal complexity and diacritic dependency as significant obstacles.<n>The findings reveal a growing body of multilingual and monolingual resources, even though the field is constrained by socio-cultural factors such as code-switching and desertion of language for digital usage.
  arXiv  Detail & Related papers  (2025-02-24T17:41:48Z)
• A Capabilities Approach to Studying Bias and Harm in Language Technologies [4.135516576952934]
  We consider fairness, bias, and inclusion in Language Technologies through the lens of the Capabilities Approach. The Capabilities Approach centers on what people are capable of achieving, given their intersectional social, political, and economic contexts. We detail the Capabilities Approach, its relationship to multilingual and multicultural evaluation, and how the framework affords meaningful collaboration with community members in defining and measuring the harms of Language Technologies.
  arXiv  Detail & Related papers  (2024-11-06T22:46:13Z)
• A Quantum-Inspired Analysis of Human Disambiguation Processes [0.0]
  In this thesis, we apply formalisms arising from foundational quantum mechanics to study ambiguities arising from linguistics. Results were subsequently used to predict human behaviour and outperformed current NLP methods.
  arXiv  Detail & Related papers  (2024-08-14T09:21:23Z)
• The Call for Socially Aware Language Technologies [94.6762219597438]
  We argue that many of these issues share a common core: a lack of awareness of the factors, context, and implications of the social environment in which NLP operates.<n>We argue that substantial challenges remain for NLP to develop social awareness and that we are just at the beginning of a new era for the field.
  arXiv  Detail & Related papers  (2024-05-03T18:12:39Z)
• From Multiple-Choice to Extractive QA: A Case Study for English and Arabic [51.13706104333848]
  We explore the feasibility of repurposing an existing multilingual dataset for a new NLP task.<n>We present annotation guidelines and a parallel EQA dataset for English and Modern Standard Arabic.<n>We aim to help others adapt our approach for the remaining 120 BELEBELE language variants, many of which are deemed under-resourced.
  arXiv  Detail & Related papers  (2024-04-26T11:46:05Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets.<n>This survey delves into an important attribute of these datasets: the dialect of a language.<n>Motivated by the performance degradation of NLP models for dialectal datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation [52.903665881174845]
  We propose a pyramid of types of knowledge required in computational argumentation. We briefly discuss the state of the art on the role and integration of these types in the field.
  arXiv  Detail & Related papers  (2021-07-01T08:12:41Z)
• Experience Grounds Language [185.73483760454454]
  Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates. Despite the incredible effectiveness of language processing models to tackle tasks after being trained on text alone, successful linguistic communication relies on a shared experience of the world.
  arXiv  Detail & Related papers  (2020-04-21T16:56:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.