Arabic Dataset for LLM Safeguard Evaluation

• URL: http://arxiv.org/abs/2410.17040v2
• Date: Sun, 09 Feb 2025 10:11:59 GMT
• Title: Arabic Dataset for LLM Safeguard Evaluation
• Authors: Yasser Ashraf, Yuxia Wang, Bin Gu, Preslav Nakov, Timothy Baldwin,
• Abstract summary: This study explores the safety of large language models (LLMs) in Arabic with its linguistic and cultural complexities. We present an Arab-region-specific safety evaluation dataset consisting of 5,799 questions, including direct attacks, indirect attacks, and harmless requests with sensitive words.
• Score: 62.96160492994489
• License:
• Abstract: The growing use of large language models (LLMs) has raised concerns regarding their safety. While many studies have focused on English, the safety of LLMs in Arabic, with its linguistic and cultural complexities, remains under-explored. Here, we aim to bridge this gap. In particular, we present an Arab-region-specific safety evaluation dataset consisting of 5,799 questions, including direct attacks, indirect attacks, and harmless requests with sensitive words, adapted to reflect the socio-cultural context of the Arab world. To uncover the impact of different stances in handling sensitive and controversial topics, we propose a dual-perspective evaluation framework. It assesses the LLM responses from both governmental and opposition viewpoints. Experiments over five leading Arabic-centric and multilingual LLMs reveal substantial disparities in their safety performance. This reinforces the need for culturally specific datasets to ensure the responsible deployment of LLMs.

Related papers

• Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts [40.0358736497799]
  Large language models (LLMs) are known to have the potential to generate harmful content. This paper introduces Qorgau, a novel dataset specifically designed for safety evaluation in Kazakh and Russian.
  arXiv  Detail & Related papers  (2025-02-19T11:33:22Z)
• LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps [63.10843814055688]
  M-ALERT is a benchmark that evaluates the safety of Large Language Models in five languages: English, French, German, Italian, and Spanish. M-ALERT includes 15k high-quality prompts per language, totaling 75k, following the detailed ALERT taxonomy.
  arXiv  Detail & Related papers  (2024-12-19T16:46:54Z)
• SafeWorld: Geo-Diverse Safety Alignment [107.84182558480859]
  We introduce SafeWorld, a novel benchmark specifically designed to evaluate Large Language Models (LLMs) SafeWorld encompasses 2,342 test user queries, each grounded in high-quality, human-verified cultural norms and legal policies from 50 countries and 493 regions/races. Our trained SafeWorldLM outperforms all competing models, including GPT-4o on all three evaluation dimensions by a large margin.
  arXiv  Detail & Related papers  (2024-12-09T13:31:46Z)
• Guardians of Discourse: Evaluating LLMs on Multilingual Offensive Language Detection [10.129235204880443]
  We evaluate the impact of different prompt languages and augmented translation data for the task in non-English contexts. We discuss the impact of the inherent bias in LLMs and the datasets in the mispredictions related to sensitive topics.
  arXiv  Detail & Related papers  (2024-10-21T04:08:16Z)
• Hate Personified: Investigating the role of LLMs in content moderation [64.26243779985393]
  For subjective tasks such as hate detection, where people perceive hate differently, the Large Language Model's (LLM) ability to represent diverse groups is unclear. By including additional context in prompts, we analyze LLM's sensitivity to geographical priming, persona attributes, and numerical information to assess how well the needs of various groups are reflected.
  arXiv  Detail & Related papers  (2024-10-03T16:43:17Z)
• Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English [1.3359598694842185]
  We explore the ability of various LLMs to comprehend the cultural aspects of two regional languages: Malayalam (state of Kerala, India) and Yoruba (West Africa) We demonstrate that although LLMs show a high cultural similarity for English, they fail to capture the cultural nuances across these 6 metrics for Malayalam and Yoruba. This will have huge implications for enhancing the user experience of chat-based LLMs and also improving the validity of large-scale LLM agent-based market research.
  arXiv  Detail & Related papers  (2024-09-14T02:21:17Z)
• Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
  Large language models (LLMs) have demonstrated substantial commonsense understanding. This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
  arXiv  Detail & Related papers  (2024-05-07T20:28:34Z)
• AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic [0.0]
  We introduce AraTrust, the first comprehensive trustworthiness benchmark for Large Language Models (LLMs) in Arabic. GPT-4 was the most trustworthy LLM, while open-source models, particularly AceGPT 7B and Jais 13B, struggled to achieve a score of 60% in our benchmark.
  arXiv  Detail & Related papers  (2024-03-14T00:45:24Z)
• The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts [46.089025223336854]
  This paper examines the variations in safety challenges faced by large language models across different languages. We compare how state-of-the-art LLMs respond to the same set of malicious prompts written in higher- vs. lower-resource languages.
  arXiv  Detail & Related papers  (2024-01-23T23:12:09Z)
• Safety Assessment of Chinese Large Language Models [51.83369778259149]
  Large language models (LLMs) may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes. To promote the deployment of safe, responsible, and ethical AI, we release SafetyPrompts including 100k augmented prompts and responses by LLMs.
  arXiv  Detail & Related papers  (2023-04-20T16:27:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.