Guidelines for Fine-grained Sentence-level Arabic Readability Annotation

• URL: http://arxiv.org/abs/2410.08674v1
• Date: Fri, 11 Oct 2024 09:59:46 GMT
• Title: Guidelines for Fine-grained Sentence-level Arabic Readability Annotation
• Authors: Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, Abdallah Abushmaes,
• Abstract summary: The Balanced Arabic Readability Evaluation Corpus (BAREC) project is designed to address the need for comprehensive Arabic language resources aligned with diverse readability levels. Inspired by the Taha/Arabi21 readability reference, BAREC aims to provide a standardized reference for assessing sentence-level Arabic text readability across 19 distinct levels. This paper focuses on our meticulous annotation guidelines, demonstrated through the analysis of 10,631 sentences/phrases (113,651 words)
• Score: 9.261022921574318
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents the foundational framework and initial findings of the Balanced Arabic Readability Evaluation Corpus (BAREC) project, designed to address the need for comprehensive Arabic language resources aligned with diverse readability levels. Inspired by the Taha/Arabi21 readability reference, BAREC aims to provide a standardized reference for assessing sentence-level Arabic text readability across 19 distinct levels, ranging in targets from kindergarten to postgraduate comprehension. Our ultimate goal with BAREC is to create a comprehensive and balanced corpus that represents a wide range of genres, topics, and regional variations through a multifaceted approach combining manual annotation with AI-driven tools. This paper focuses on our meticulous annotation guidelines, demonstrated through the analysis of 10,631 sentences/phrases (113,651 words). The average pairwise inter-annotator agreement, measured by Quadratic Weighted Kappa, is 79.9%, reflecting a high level of substantial agreement. We also report competitive results for benchmarking automatic readability assessment. We will make the BAREC corpus and guidelines openly accessible to support Arabic language research and education.

Related papers

• A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment [10.853984915007961]
  This paper introduces the Balanced Arabic Readability Evaluation Corpus BAREC. BAREC consists of 68,182 sentences spanning 1+ million words, carefully curated to cover 19 readability levels. The corpus was fully manually annotated by a large team of annotators.
  arXiv  Detail & Related papers  (2025-02-19T08:16:11Z)
• Strategies for Arabic Readability Modeling [9.976720880041688]
  Automatic readability assessment is relevant to building NLP applications for education, content analysis, and accessibility. We present a set of experimental results on Arabic readability assessment using a diverse range of approaches.
  arXiv  Detail & Related papers  (2024-07-03T11:54:11Z)
• ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA [50.90538760832107]
  This research presents a novel task, Review-Level Opinion Aspect Sentiment Target (ROAST) ROAST seeks to close the gap between sentence-level and text-level ABSA by identifying every ABSA constituent at the review level. We extend the available datasets to enable ROAST, addressing the drawbacks noted in previous research.
  arXiv  Detail & Related papers  (2024-05-30T17:29:15Z)
• The SAMER Arabic Text Simplification Corpus [9.369209124775043]
  SAMER Corpus is the first manually annotated Arabic parallel corpus for text simplification targeting school-aged learners. Our corpus comprises texts of 159K words selected from 15 publicly available Arabic fiction novels published between 1865 and 1955. Our corpus includes readability level annotations at both the document and word levels, as well as two simplified parallel versions for each text targeting learners at two different readability levels.
  arXiv  Detail & Related papers  (2024-04-29T11:34:06Z)
• From Multiple-Choice to Extractive QA: A Case Study for English and Arabic [51.13706104333848]
  We explore the feasibility of repurposing an existing multilingual dataset for a new NLP task. We present annotation guidelines and a parallel EQA dataset for English and Modern Standard Arabic. We aim to help others adapt our approach for the remaining 120 BELEBELE language variants, many of which are deemed under-resourced.
  arXiv  Detail & Related papers  (2024-04-26T11:46:05Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• AceGPT, Localizing Large Language Models in Arabic [73.39989503874634]
  The paper proposes a comprehensive solution that includes pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.
  arXiv  Detail & Related papers  (2023-09-21T13:20:13Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• ORCA: A Challenging Benchmark for Arabic Language Understanding [8.9379057739817]
  ORCA is a publicly available benchmark for Arabic language understanding evaluation. To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models.
  arXiv  Detail & Related papers  (2022-12-21T04:35:43Z)
• CEFR-Based Sentence Difficulty Annotation and Assessment [25.71796445061561]
  The CEFR-based Sentence Profile (CEFR-SP) corpus contains 17k English sentences annotated with the levels based on the Common European Framework of Reference for Languages. In the experiments in this study, our method achieved a macro-F1 score of 84.5% in the level assessment, thus outperforming strong baselines employed in readability assessment.
  arXiv  Detail & Related papers  (2022-10-21T07:03:30Z)
• CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark [144.05723617401674]
  General-purpose language intelligence evaluation has been a longstanding goal for natural language processing. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. We propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features.
  arXiv  Detail & Related papers  (2021-12-27T11:08:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.