HalluScore: Large Language Model Hallucination Question Answering Benchmark

• URL: http://arxiv.org/abs/2605.17007v1
• Date: Sat, 16 May 2026 14:08:15 GMT
• Title: HalluScore: Large Language Model Hallucination Question Answering Benchmark
• Abstract summary: HalluScore is a structured Arabic question answering benchmark designed to evaluate hallucination behavior in large language models.<n>It contains 827 carefully curated questions for evaluating, detecting, and mitigating hallucination in LLMs.<n>We conduct a comprehensive empirical analysis of hallucination patterns across 17 Arabic, multilingual, and reasoning LLMs.
• Score: 3.8100688074986095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have achieved remarkable progress in natural language generation, but remain susceptible to hallucination. In response to growing concerns about hallucinations, several benchmarks have been developed, primarily in English and Chinese. However, Arabic remains underrepresented, with limited benchmarks for LLMs hallucination due to scarce annotated resources and the language's morphological complexity. Consequently, existing benchmarks do not adequately reflect the linguistic, cultural, and reasoning characteristics of Arabic. To address this gap, we introduce HalluScore, a structured Arabic question answering benchmark designed to evaluate hallucination behavior in LLMs across different levels of reasoning difficulty, various knowledge domains, historical timelines, and culturally grounded Arabic scenarios. It contains 827 carefully curated questions for evaluating, detecting, and mitigating hallucination in LLMs. The dataset was constructed through a structured pipeline involving quality assurance, filtering for clarity and factual validity, and model-driven selection to retain questions that consistently trigger hallucinations. Each question is linked to verified ground-truth evidence, answer explanations, and multi-label annotations. Using the HalluScore benchmark, we conduct a comprehensive empirical analysis of hallucination patterns across 17 Arabic, multilingual, and reasoning LLMs. Moreover, we provide high-quality human annotations identifying hallucinated, non-hallucinated, and partially hallucinated responses of all evaluated LLMs. These results suggest that hallucination in Arabic LLMs extends beyond factual inaccuracies, encompassing challenges related to cultural understanding, linguistic reasoning, and logical consistency. We release HalluScore to support future research on improving the reliability and cultural competence of LLMs in Arabic.

Related papers

• Halluverse-M^3: A multitask multilingual benchmark for hallucination in LLMs [2.453830698820308]
  Halluverse-M3 is a dataset designed to enable systematic analysis of hallucinations across multiple languages.<n>The dataset explicitly distinguishes between entity-level, relation-level, and sentence-level hallucinations.<n>Halluverse-M3 provides a realistic and challenging benchmark for studying hallucinations in multilingual, multi-task settings.
  arXiv  Detail & Related papers  (2026-02-06T18:16:09Z)
• AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs [3.8100688074986095]
  This paper presents the first hallucination evaluation of Arabic and multilingual large language models (LLMs)<n>This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models.<n>Results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks.
  arXiv  Detail & Related papers  (2025-09-04T20:57:35Z)
• HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations [2.3732122943029164]
  We introduce HalluVerse25, a multilingual dataset that categorizes fine-grained hallucinations in English, Arabic, and Turkish.<n>Our dataset construction pipeline uses an LLM to inject hallucinations into factual biographical sentences, followed by a rigorous human annotation process to ensure data quality.
  arXiv  Detail & Related papers  (2025-03-10T20:24:07Z)
• How Much Do LLMs Hallucinate across Languages? On Multilingual Estimation of LLM Hallucination in the Wild [22.24685025857759]
  hallucination is the tendency of Large Language Models to generate non-factual or unfaithful responses.<n>We train a multilingual hallucination detection model and conduct a large-scale study across 30 languages.<n>We find that while LLMs generate longer responses with more hallucinated tokens for higher-resource languages, there is no correlation between length-normalized hallucination rates of languages and their digital representation.
  arXiv  Detail & Related papers  (2025-02-18T11:32:43Z)
• Multilingual Hallucination Gaps in Large Language Models [5.505634045241288]
  We study the phenomenon of hallucinations across multiple languages in freeform text generation. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. Our results reveal variations in hallucination rates, especially between high and low resource languages.
  arXiv  Detail & Related papers  (2024-10-23T20:41:51Z)
• LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models [96.64960606650115]
  LongHalQA is an LLM-free hallucination benchmark that comprises 6K long and complex hallucination text. LongHalQA is featured by GPT4V-generated hallucinatory data that are well aligned with real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-13T18:59:58Z)
• WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries [64.239202960816]
  We introduce WildHallucinations, a benchmark that evaluates factuality. It does so by prompting large language models to generate information about entities mined from user-chatbot conversations in the wild. We evaluate 118,785 generations from 15 LLMs on 7,919 entities.
  arXiv  Detail & Related papers  (2024-07-24T17:59:05Z)
• Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models [70.19081534515371]
  Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks. They generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. We propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers.
  arXiv  Detail & Related papers  (2024-07-04T18:47:42Z)
• ANAH: Analytical Annotation of Hallucinations in Large Language Models [65.12177400764506]
  We present $textbfANAH$, a dataset that offers $textbfAN$alytical $textbfA$nnotation of hallucinations in Large Language Models. ANAH consists of 12k sentence-level annotations for 4.3k LLM responses covering over 700 topics, constructed by a human-in-the-loop pipeline. Thanks to the fine granularity of the hallucination annotations, we can quantitatively confirm that the hallucinations of LLMs accumulate in the answer and use ANAH to train and evaluate hallucination annotators.
  arXiv  Detail & Related papers  (2024-05-30T17:54:40Z)
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models [134.6697160940223]
  hallucination poses great challenge to trustworthy and reliable deployment of large language models. Three key questions should be well studied: how to detect hallucinations (detection), why do LLMs hallucinate (source), and what can be done to mitigate them. This work presents a systematic empirical study on LLM hallucination, focused on the the three aspects of hallucination detection, source and mitigation.
  arXiv  Detail & Related papers  (2024-01-06T12:40:45Z)
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models [146.87696738011712]
  Large language models (LLMs) are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge. To understand what types of content and to which extent LLMs are apt to hallucinate, we introduce the Hallucination Evaluation benchmark for Large Language Models (HaluEval)
  arXiv  Detail & Related papers  (2023-05-19T15:36:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.