AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs

• URL: http://arxiv.org/abs/2509.04656v2
• Date: Tue, 09 Sep 2025 13:49:46 GMT
• Title: AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
• Abstract summary: This paper presents the first hallucination evaluation of Arabic and multilingual large language models (LLMs)<n>This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models.<n>Results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks.
• Score: 3.8100688074986095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, extensive research on the hallucination of the large language models (LLMs) has mainly focused on the English language. Despite the growing number of multilingual and Arabic-specific LLMs, evaluating LLMs' hallucination in the Arabic context remains relatively underexplored. The knowledge gap is particularly pressing given Arabic's widespread use across many regions and its importance in global communication and media. This paper presents the first comprehensive hallucination evaluation of Arabic and multilingual LLMs on two critical Arabic natural language generation tasks: generative question answering (GQA) and summarization. This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models. To assess the factual consistency and faithfulness of LLMs' outputs, we developed a fine-grained hallucination evaluation framework consisting of 12 fine-grained hallucination indicators that represent the varying characteristics of each task. The results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks. Notably, the Arabic pre-trained model Allam consistently demonstrates lower hallucination rates than multilingual models and a comparative performance with reasoning-based models. The code is available at: https://github.com/aishaalansari57/AraHalluEval

Related papers

• HalluScore: Large Language Model Hallucination Question Answering Benchmark [3.8100688074986095]
  HalluScore is a structured Arabic question answering benchmark designed to evaluate hallucination behavior in large language models.<n>It contains 827 carefully curated questions for evaluating, detecting, and mitigating hallucination in LLMs.<n>We conduct a comprehensive empirical analysis of hallucination patterns across 17 Arabic, multilingual, and reasoning LLMs.
  arXiv  Detail & Related papers  (2026-05-16T14:08:15Z)
• Halluverse-M^3: A multitask multilingual benchmark for hallucination in LLMs [2.453830698820308]
  Halluverse-M3 is a dataset designed to enable systematic analysis of hallucinations across multiple languages.<n>The dataset explicitly distinguishes between entity-level, relation-level, and sentence-level hallucinations.<n>Halluverse-M3 provides a realistic and challenging benchmark for studying hallucinations in multilingual, multi-task settings.
  arXiv  Detail & Related papers  (2026-02-06T18:16:09Z)
• DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models [54.10223256792762]
  We present DialectalArabicMMLU, a new benchmark for evaluating the performance of large language models (LLMs) across Arabic dialects.<n>We extend the MMLU-Redux framework through manual translation and adaptation of 3K multiple-choice question-answer pairs into five major dialects.
  arXiv  Detail & Related papers  (2025-10-31T15:17:06Z)
• PerHalluEval: Persian Hallucination Evaluation Benchmark for Large Language Models [2.049767929976436]
  PerHalluEval is the first dynamic hallucination evaluation benchmark tailored for the Persian language.<n>Our benchmark generates plausible answers and summaries regarding QA and summarization tasks.<n>We used the log probabilities of generated tokens to select the most believable hallucinated instances.
  arXiv  Detail & Related papers  (2025-09-25T12:50:46Z)
• HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations [2.3732122943029164]
  We introduce HalluVerse25, a multilingual dataset that categorizes fine-grained hallucinations in English, Arabic, and Turkish.<n>Our dataset construction pipeline uses an LLM to inject hallucinations into factual biographical sentences, followed by a rigorous human annotation process to ensure data quality.
  arXiv  Detail & Related papers  (2025-03-10T20:24:07Z)
• How Much Do LLMs Hallucinate across Languages? On Multilingual Estimation of LLM Hallucination in the Wild [11.82100047858478]
  hallucination is the tendency of Large Language Models to generate non-factual or unfaithful responses.<n>We train a multilingual hallucination detection model and conduct a large-scale study across 30 languages.<n>We find that while LLMs generate longer responses with more hallucinated tokens for higher-resource languages, there is no correlation between length-normalized hallucination rates of languages and their digital representation.
  arXiv  Detail & Related papers  (2025-02-18T11:32:43Z)
• FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs [2.871226288151562]
  This paper introduces FaithBench, a summarization hallucination benchmark comprising challenging hallucinations made by 10 modern LLMs. Our results show GPT-4o and GPT-3.5-Turbo produce the least hallucinations. Even the best hallucination detection models have near 50% accuracies on FaithBench, indicating lots of room for future improvement.
  arXiv  Detail & Related papers  (2024-10-17T04:30:46Z)
• LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models [96.64960606650115]
  LongHalQA is an LLM-free hallucination benchmark that comprises 6K long and complex hallucination text. LongHalQA is featured by GPT4V-generated hallucinatory data that are well aligned with real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-13T18:59:58Z)
• VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models [59.05674402770661]
  This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs) VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis.
  arXiv  Detail & Related papers  (2024-06-24T06:21:59Z)
• ANAH: Analytical Annotation of Hallucinations in Large Language Models [65.12177400764506]
  We present $textbfANAH$, a dataset that offers $textbfAN$alytical $textbfA$nnotation of hallucinations in Large Language Models. ANAH consists of 12k sentence-level annotations for 4.3k LLM responses covering over 700 topics, constructed by a human-in-the-loop pipeline. Thanks to the fine granularity of the hallucination annotations, we can quantitatively confirm that the hallucinations of LLMs accumulate in the answer and use ANAH to train and evaluate hallucination annotators.
  arXiv  Detail & Related papers  (2024-05-30T17:54:40Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• "Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation [90.09260023184932]
  Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. NoMIRACL is a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. We measure relevance assessment using: (i) hallucination rate, measuring model tendency to hallucinate, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
  arXiv  Detail & Related papers  (2023-12-18T17:18:04Z)
• Evaluating Hallucinations in Chinese Large Language Models [65.4771562909392]
  We establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. We consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated.
  arXiv  Detail & Related papers  (2023-10-05T07:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.