IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

• URL: http://arxiv.org/abs/2603.23750v1
• Date: Tue, 24 Mar 2026 22:18:16 GMT
• Title: IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge
• Abstract summary: We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran, Hadith, and Fiqh.<n>Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge.<n>The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs.
• Score: 6.894744675083236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8\% to 93.8\% (by Gemini 3 Flash). The Quran track shows the widest span (99.3\% to 32.4\%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.

Related papers

• Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI [0.0]
  Large language models (LLMs) are increasingly used for knowledge-intensive question answering.<n>This survey reviews the emerging field of Islamic LLMs and trustworthy Islamic AI.
  arXiv  Detail & Related papers  (2026-06-15T12:22:36Z)
• Benchmarks in Leipzig [62.700298058731505]
  Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers.<n>We evaluate these questions in three stages: a single attempt by five state-of-the-art LLMs, a 20-runs-per-model evaluation, and a 3-run attempt with two heavy-thinking models.<n>After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions.
  arXiv  Detail & Related papers  (2026-06-04T07:59:08Z)
• IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions [1.3052252174353483]
  IslamicLegalBench is the first benchmark evaluating LLMs across seven schools of Islamic jurisprudence.<n>Best model achieves only 68% correctness with 21% hallucination.<n>Few-shot prompting provides minimal gains, improving only 2 of 9 models by >1%.
  arXiv  Detail & Related papers  (2026-02-02T10:30:59Z)
• DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models [54.10223256792762]
  We present DialectalArabicMMLU, a new benchmark for evaluating the performance of large language models (LLMs) across Arabic dialects.<n>We extend the MMLU-Redux framework through manual translation and adaptation of 3K multiple-choice question-answer pairs into five major dialects.
  arXiv  Detail & Related papers  (2025-10-31T15:17:06Z)
• Can LLMs Write Faithfully? An Agent-Based Evaluation of LLM-generated Islamic Content [1.922162958936778]
  Large language models are increasingly used for Islamic guidance, but risk misquoting texts, misapplying jurisprudence, or producing culturally inconsistent responses.<n>We pilot an evaluation of GPT-4o, Ansari AI, and Fanar on prompts from authentic Islamic blogs.<n> GPT-4o scored highest in Islamic Accuracy (3.93) and Citation (3.38), Ansari AI followed (3.68, 3.32), and Fanar lagged (2.76, 1.82)
  arXiv  Detail & Related papers  (2025-10-28T14:05:55Z)
• Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions [10.53116395328794]
  We introduce a novel benchmark FiqhQA focused on the LLM generated Islamic rulings explicitly categorized by the four major Sunni schools of thought, in both Arabic and English.<n>Our zero-shot and abstention experiments reveal significant variation across LLMs, languages, and legal schools of thought.<n>To the best of our knowledge, this is the first study to benchmark the efficacy of LLMs for fine-grained Islamic school of thought specific ruling generation and to evaluate abstention for Islamic queries.
  arXiv  Detail & Related papers  (2025-08-04T07:27:26Z)
• HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning [56.221060995324436]
  Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning.<n>Do these models truly understand commonsense knowledge, or just memorize expression patterns?<n>We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases.
  arXiv  Detail & Related papers  (2025-02-17T03:24:02Z)
• Humanity's Last Exam [521.5879252045291]
  Humanity's Last Exam (HLE) is a multi-modal benchmark at the frontier of human knowledge.<n>It consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences.<n>Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval.
  arXiv  Detail & Related papers  (2025-01-24T05:27:46Z)
• TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish [54.51310112013655]
  We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models.
  arXiv  Detail & Related papers  (2024-07-17T08:28:55Z)
• The Earth is Flat? Unveiling Factual Errors in Large Language Models [89.94270049334479]
  Large Language Models (LLMs) like ChatGPT are in various applications due to their extensive knowledge from pre-training and fine-tuning. Despite this, they are prone to generating factual and commonsense errors, raising concerns in critical areas like healthcare, journalism, and education. We introduce a novel, automatic testing framework, FactChecker, aimed at uncovering factual inaccuracies in LLMs.
  arXiv  Detail & Related papers  (2024-01-01T14:02:27Z)
• Building Domain-Specific LLMs Faithful To The Islamic Worldview: Mirage or Technical Possibility? [0.0]
  Large Language Models (LLMs) have demonstrated remarkable performance across numerous natural language understanding use cases. In the context of Islam and its representation, accurate and factual representation of its beliefs and teachings rooted in the Quran and Sunnah is key. This work focuses on the challenge of building domain-specific LLMs faithful to the Islamic worldview.
  arXiv  Detail & Related papers  (2023-12-11T18:59:09Z)
• QASiNa: Religious Domain Question Answering using Sirah Nabawiyah [0.0]
  In Islam we strictly regulate the sources of information and who can give interpretations or tafseer for that sources. The approach used by LLM to generate answers based on its own interpretation is similar to the concept of tafseer. We propose the Question Answering Sirah Nabawiyah (QASiNa) dataset, a novel dataset compiled from Sirah Nabawiyah literatures in Indonesian language.
  arXiv  Detail & Related papers  (2023-10-12T07:52:19Z)
• Statistical Knowledge Assessment for Large Language Models [79.07989821512128]
  Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? We propose KaRR, a statistical approach to assess factual knowledge for LLMs. Our results reveal that the knowledge in LLMs with the same backbone architecture adheres to the scaling law, while tuning on instruction-following data sometimes compromises the model's capability to generate factually correct text reliably.
  arXiv  Detail & Related papers  (2023-05-17T18:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.