Guidelines for Fine-grained Sentence-level Arabic Readability Annotation

• URL: http://arxiv.org/abs/2410.08674v1
• Date: Fri, 11 Oct 2024 09:59:46 GMT
• Title: Guidelines for Fine-grained Sentence-level Arabic Readability Annotation
• Authors: Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, Abdallah Abushmaes,
• Abstract summary: The Balanced Arabic Readability Evaluation Corpus (BAREC) project is designed to address the need for comprehensive Arabic language resources aligned with diverse readability levels. Inspired by the Taha/Arabi21 readability reference, BAREC aims to provide a standardized reference for assessing sentence-level Arabic text readability across 19 distinct levels. This paper focuses on our meticulous annotation guidelines, demonstrated through the analysis of 10,631 sentences/phrases (113,651 words)
• Score: 9.261022921574318
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents the foundational framework and initial findings of the Balanced Arabic Readability Evaluation Corpus (BAREC) project, designed to address the need for comprehensive Arabic language resources aligned with diverse readability levels. Inspired by the Taha/Arabi21 readability reference, BAREC aims to provide a standardized reference for assessing sentence-level Arabic text readability across 19 distinct levels, ranging in targets from kindergarten to postgraduate comprehension. Our ultimate goal with BAREC is to create a comprehensive and balanced corpus that represents a wide range of genres, topics, and regional variations through a multifaceted approach combining manual annotation with AI-driven tools. This paper focuses on our meticulous annotation guidelines, demonstrated through the analysis of 10,631 sentences/phrases (113,651 words). The average pairwise inter-annotator agreement, measured by Quadratic Weighted Kappa, is 79.9%, reflecting a high level of substantial agreement. We also report competitive results for benchmarking automatic readability assessment. We will make the BAREC corpus and guidelines openly accessible to support Arabic language research and education.

Related papers

• Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents [61.41316121093604]
  We present InsCoQA, a novel benchmark for evaluating large language models (LLMs) in the context of conversational question answering (CQA) Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents. We also propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
  arXiv  Detail & Related papers  (2024-10-01T09:10:00Z)
• Strategies for Arabic Readability Modeling [9.976720880041688]
  Automatic readability assessment is relevant to building NLP applications for education, content analysis, and accessibility. We present a set of experimental results on Arabic readability assessment using a diverse range of approaches.
  arXiv  Detail & Related papers  (2024-07-03T11:54:11Z)
• ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA [50.90538760832107]
  This research presents a novel task, Review-Level Opinion Aspect Sentiment Target (ROAST) ROAST seeks to close the gap between sentence-level and text-level ABSA by identifying every ABSA constituent at the review level. We extend the available datasets to enable ROAST, addressing the drawbacks noted in previous research.
  arXiv  Detail & Related papers  (2024-05-30T17:29:15Z)
• The SAMER Arabic Text Simplification Corpus [9.369209124775043]
  SAMER Corpus is the first manually annotated Arabic parallel corpus for text simplification targeting school-aged learners. Our corpus comprises texts of 159K words selected from 15 publicly available Arabic fiction novels published between 1865 and 1955. Our corpus includes readability level annotations at both the document and word levels, as well as two simplified parallel versions for each text targeting learners at two different readability levels.
  arXiv  Detail & Related papers  (2024-04-29T11:34:06Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• AceGPT, Localizing Large Language Models in Arabic [73.39989503874634]
  The paper proposes a comprehensive solution that includes pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.
  arXiv  Detail & Related papers  (2023-09-21T13:20:13Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• ORCA: A Challenging Benchmark for Arabic Language Understanding [8.9379057739817]
  ORCA is a publicly available benchmark for Arabic language understanding evaluation. To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models.
  arXiv  Detail & Related papers  (2022-12-21T04:35:43Z)
• CEFR-Based Sentence Difficulty Annotation and Assessment [25.71796445061561]
  The CEFR-based Sentence Profile (CEFR-SP) corpus contains 17k English sentences annotated with the levels based on the Common European Framework of Reference for Languages. In the experiments in this study, our method achieved a macro-F1 score of 84.5% in the level assessment, thus outperforming strong baselines employed in readability assessment.
  arXiv  Detail & Related papers  (2022-10-21T07:03:30Z)
• CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark [144.05723617401674]
  General-purpose language intelligence evaluation has been a longstanding goal for natural language processing. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. We propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features.
  arXiv  Detail & Related papers  (2021-12-27T11:08:58Z)
• Simple or Complex? Learning to Predict Readability of Bengali Texts [6.860272388539321]
  We present a readability analysis tool capable of analyzing text written in the Bengali language. Despite being the 7th most spoken language in the world with 230 million native speakers, Bengali suffers from a lack of fundamental resources for natural language processing.
  arXiv  Detail & Related papers  (2020-12-09T01:41:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.