ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification

• URL: http://arxiv.org/abs/2608.01291v1
• Date: Sun, 02 Aug 2026 15:00:58 GMT
• Title: ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification
• Abstract summary: We present a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties.<n>The dataset is annotated with dialect labels and seven fine-grained harm categories.<n> Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance.
• Score: 1.4665143197624044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties: Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan. The dataset is annotated with dialect labels and seven fine-grained harm categories. We introduce a dual-task evaluation framework for binary safe/unsafe detection and granular harm classification across dialects. Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance, with Macro-F1 scores of 0.95 for binary classification and 0.90 for granular classification, substantially outperforming prompted frontier LLMs, including Arabic-specialized models. Our analyses show that dialect conditioning is most effective when integrated at the representation level, while significant performance gaps remain for low-resource Maghrebi dialects. We further evaluate seven frontier LLMs as response generators on harmful dialectal Arabic prompts and observe unsafe generation rates below 5 percent across models. We release the dataset and code upon acceptance to support future research on dialect-aware Arabic safety evaluation. Warning: This paper contains examples of harmful and potentially offensive content included solely for research purposes.

Related papers

• EDRAC: Benchmarking Arabic Dialect Reading Comprehension [43.861708347921336]
  We introduce EDRAC, the first large-scale benchmark for dialectal Arabic machine reading comprehension (MRC) and generative QA.<n> EDRAC contains 499 passages derived from naturally occurring spoken interactions and 4,977 corresponding QA pairs.<n>Results reveal substantial gaps between semantic answer quality and dialectal fidelity.
  arXiv  Detail & Related papers  (2026-09-01T11:51:49Z)
• SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs [56.34354368047577]
  Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts.<n>We introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios.<n>We benchmark a broad range of state-of-the-art LLMs on SurakshaEval, establish baseline safety performance, and identify recurring failure modes.
  arXiv  Detail & Related papers  (2026-08-08T02:03:33Z)
• Evaluation of Adversarial Robustness in Arabic Language Models [2.1665689529884697]
  This study aims to assess the robustness of five state-of-the-art Arabic Language Models under a distinct set of Arabic adversarial attacks.<n> insertion of diacritics can reduce the accuracy of some models by 92% while maintaining a low distance.<n>For word-level attacks, manipulating Arabic conjunctions preserves high semantic similarity scores, low distance, and leads to an accuracy degradation of up to 58%.<n>For sentence-level attacks, paraphrasing proves its effectiveness by an average reduction of 76% in the victim models' performance.
  arXiv  Detail & Related papers  (2026-07-28T14:58:28Z)
• MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection [0.7039739305668745]
  This study proposes a two-phase framework for Arabic mental health text classification.<n>In phase 1, three Arabic pre-trained language models undergo Domain-Adaptive and Task-Adaptive Pretraining.<n>In phase 2, the selected model is assessed across four configurations with full fine-tuning and Low-Rank Adaptation.
  arXiv  Detail & Related papers  (2026-06-10T20:20:41Z)
• Schützen: Evaluating LLM Safety in Bulgarian and German Contexts [53.865251738592605]
  This paper introduces Schtzen: a German--Bulgarian safety dataset designed to assess model answerability under risk.<n> Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior.
  arXiv  Detail & Related papers  (2026-06-09T18:01:19Z)
• LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety [88.98698230989186]
  Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities in low-resource languages.<n>We attribute this gap to a mismatch between language-agnostic semantic understanding ability and language-dominant safety alignment biased toward high-resource languages.<n>We propose Language-Agnostic Semantic Alignment (LASA), which anchors safety alignment directly in semantic bottlenecks.
  arXiv  Detail & Related papers  (2026-04-13T15:59:50Z)
• SalamahBench: Toward Standardized Safety Evaluation for Arabic Language Models [6.0860786631767185]
  This paper introduces SalamaBench, a unified benchmark for evaluating the safety of Arabic Language Models (ALMs)<n>Using this benchmark, we evaluate five state-of-the-art ALMs, including Fanar 1 and 2, ALLaM 2, Falcon H1R, and Jais 2, under multiple safeguard configurations.
  arXiv  Detail & Related papers  (2026-02-03T12:13:42Z)
• OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models [54.80460603255789]
  We introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era.<n>OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all systematically annotated across nine critical content risk categories.<n>In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS), a novel metric designed to model and assess overlapping and correlated content risks across different categories.
  arXiv  Detail & Related papers  (2025-11-13T13:18:27Z)
• Dhati+: Fine-tuned Large Language Models for Arabic Subjectivity Evaluation [0.0]
  Despite its significance, Arabic faces the challenge of being under-resourced.<n>The scarcity of large annotated datasets hampers the development of accurate tools for subjectivity analysis in Arabic.<n>Recent advances in deep learning and Transformers have proven highly effective for text classification in English and French.
  arXiv  Detail & Related papers  (2025-08-27T15:20:12Z)
• Lemmatization as a Classification Task: Results from Arabic across Multiple Genres [9.896789483253189]
  Lemmatization is crucial for NLP tasks in morphologically rich languages with ambiguous orthography like Arabic.<n>This paper introduces two novel approaches that frame lemmatization as classification into a Lemma-POS-Gloss (LPG) tagset.<n>We also present a new Arabic lemmatization test set covering diverse genres, standardized alongside existing datasets.
  arXiv  Detail & Related papers  (2025-06-23T08:34:33Z)
• Arabic Dataset for LLM Safeguard Evaluation [62.96160492994489]
  This study explores the safety of large language models (LLMs) in Arabic with its linguistic and cultural complexities.<n>We present an Arab-region-specific safety evaluation dataset consisting of 5,799 questions, including direct attacks, indirect attacks, and harmless requests with sensitive words.
  arXiv  Detail & Related papers  (2024-10-22T14:12:43Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• AceGPT, Localizing Large Language Models in Arabic [73.39989503874634]
  The paper proposes a comprehensive solution that includes pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.
  arXiv  Detail & Related papers  (2023-09-21T13:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.