YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models

• URL: http://arxiv.org/abs/2609.10153v1
• Date: Wed, 09 Sep 2026 13:31:56 GMT
• Title: YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models
• Abstract summary: We introduce YallaMorph, a large-scale benchmark for Arabic morphological generation.<n>We evaluate multilingual and Arabic-oriented LLMs under diacritized and undiacritized settings over 600K benchmark entries.<n>Results show that Arabic morphological generation remains difficult, especially for cliticized, unseen, and morphologically rare forms.
• Score: 12.343629326707733
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Arabic morphology remains challenging for large language models, since fluent generation does not guarantee accurate morphosyntactic control. Existing Arabic evaluations mainly target downstream tasks and do not directly test controlled morphological generation from explicit lexical and feature-based input. We introduce YallaMorph, a large-scale benchmark for Arabic morphological generation covering verbs, nouns, adjectives, their cliticized forms, and invalid configurations. We evaluate multilingual and Arabic-oriented LLMs under diacritized and undiacritized settings over 600K benchmark entries. Results show that Arabic morphological generation remains difficult, especially for cliticized, unseen, and morphologically rare forms.

Related papers

• Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models [13.143003615122245]
  Large language models (LLMs) perform strongly on many NLP tasks, but their ability to produce explicit linguistic structure remains unclear.<n>We evaluate instruction-tuned LLMs on two structured prediction tasks for Standard Arabic: morphosyntactic tagging and labeled dependency parsing.
  arXiv  Detail & Related papers  (2026-03-17T16:06:29Z)
• Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs [6.520393736113561]
  This work investigates how effectively large language models (LLMs) represent and generate Arabic root-pattern morphology.<n>Our study begins with an evaluation of morphological fidelity across Arabic and multilingual tokenizers against gold-standard segmentation.<n>Our findings reveal that tokenizer morphological alignment is not necessary nor sufficient for morphological generation, which questions the role of morphological tokenization in downstream performance.
  arXiv  Detail & Related papers  (2026-03-16T18:05:01Z)
• IMPACT: Inflectional Morphology Probes Across Complex Typologies [0.0]
  IMPACT is a synthetically generated evaluation framework focused on inflectional morphology.<n>It is designed to evaluate performance across five morphologically rich languages: Arabic, Russian, Finnish, Turkish, and Hebrew.<n>We assess eight multilingual LLMs that, despite strong English performance, struggle with other languages and uncommon morphological patterns.
  arXiv  Detail & Related papers  (2025-06-30T14:58:23Z)
• Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion [70.23624194206171]
  This paper addresses the need for democratizing large language models (LLM) in the Arab world.<n>One practical objective for an Arabic LLM is to utilize an Arabic-specific vocabulary for the tokenizer that could speed up decoding.<n>Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion.
  arXiv  Detail & Related papers  (2024-12-16T19:29:06Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• AceGPT, Localizing Large Language Models in Arabic [73.39989503874634]
  The paper proposes a comprehensive solution that includes pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.
  arXiv  Detail & Related papers  (2023-09-21T13:20:13Z)
• Graphemic Normalization of the Perso-Arabic Script [47.429213930688086]
  This paper documents the challenges that Perso-Arabic presents beyond the best-documented languages. We focus on the situation in natural language processing (NLP), which is affected by multiple, often neglected, issues. We evaluate the effects of script normalization on eight languages from diverse language families in the Perso-Arabic script diaspora on machine translation and statistical language modeling tasks.
  arXiv  Detail & Related papers  (2022-10-21T21:59:44Z)
• Post-hoc analysis of Arabic transformer models [20.741730718486032]
  We probe how linguistic information is encoded in the transformer models, trained on different Arabic dialects. We perform a layer and neuron analysis on the models using morphological tagging tasks for different dialects of Arabic and a dialectal identification task.
  arXiv  Detail & Related papers  (2022-10-18T16:53:51Z)
• UniMorph 4.0: Universal Morphology [104.69846084893298]
  This paper presents the expansions and improvements made on several fronts over the last couple of years. Collaborative efforts by numerous linguists have added 67 new languages, including 30 endangered languages. In light of the last UniMorph release, we also augmented the database with morpheme segmentation for 16 languages.
  arXiv  Detail & Related papers  (2022-05-07T09:19:02Z)
• Interpreting Arabic Transformer Models [18.98681439078424]
  We probe how linguistic information is encoded in Arabic pretrained models, trained on different varieties of Arabic language. We perform a layer and neuron analysis on the models using three intrinsic tasks: two morphological tagging tasks based on MSA (modern standard Arabic) and dialectal POS-tagging and a dialectal identification task.
  arXiv  Detail & Related papers  (2022-01-19T06:32:25Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)
• A Simple Joint Model for Improved Contextual Neural Lemmatization [60.802451210656805]
  We present a simple joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages. Our paper describes the model in addition to training and decoding procedures.
  arXiv  Detail & Related papers  (2019-04-04T02:03:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.