IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

• URL: http://arxiv.org/abs/2608.04703v1
• Date: Wed, 05 Aug 2026 11:12:48 GMT
• Title: IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)
• Abstract summary: IslamicTurathBench (ISTB) is a multi-task, multi-discipline dataset for evaluating large language models (LLMs)<n>ISTB contains 3,465 question-answer items drawn from 35 recognised source works spanning more than 12 centuries of scholarship across seven key fields of Islamic Studies.
• Score: 0.16060719742433224
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used for question answering, education, and research, including in religious and cultural domains where answers depend on specialised source traditions. Yet in Islamic Studies, key concepts, methods, and debates preserved in the authoritative scholarly tradition, known as turath, lack high-quality annotated resources. We introduce IslamicTurathBench (ISTB), a multi-task, multi-discipline dataset for evaluating LLMs on classical Islamic scholarship. Developed and reviewed by domain experts, ISTB contains 3,465 question-answer items drawn from 35 recognised source works spanning more than 12 centuries of scholarship across seven key fields of Islamic Studies. To enable comprehensive profiling of model capabilities, ISTB is structured along two axes: scholarly demand (Beginner, Intermediate, and Advanced) and task format (multiple-choice questions, passage-based comprehension, and open-ended knowledge questions). ISTB includes aggregated scores from a scholarly human reference panel and zero-shot baselines from ten systems. The dataset supports reproducible evaluation of language-model behaviour across source works, disciplines, scholarly-demand levels, and question formats in a historically layered scholarly domain.

Related papers

• Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI [0.0]
  Large language models (LLMs) are increasingly used for knowledge-intensive question answering.<n>This survey reviews the emerging field of Islamic LLMs and trustworthy Islamic AI.
  arXiv  Detail & Related papers  (2026-06-15T12:22:36Z)
• Mubeen AI: A Specialized Arabic Language Model for Heritage Preservation and User Intent Understanding [0.0]
  Mubeen is a proprietary Arabic language model developed by MASARAT SA.<n>It is optimized for deep understanding of Arabic linguistics, Islamic studies, and cultural heritage.<n>The model incorporates seminal scholarly works in linguistics, jurisprudence, hadith, and Quranic exegesis.
  arXiv  Detail & Related papers  (2025-10-27T12:29:27Z)
• SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala [39.525952729268994]
  We introduce SinhalaMMLU, the first multiple-choice question answering benchmark designed specifically for Sinhala.<n>The dataset includes over 7,000 questions spanning secondary to collegiate education levels, aligned with the Sri Lankan national curriculum.<n>We evaluate 26 LLMs on SinhalaMMLU and observe that, while Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies at 67% and 62% respectively, overall model performance remains limited.
  arXiv  Detail & Related papers  (2025-09-03T09:22:39Z)
• MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams [50.293164501645975]
  Multimodal large language models (MLLMs) integrate language and visual cues for problem-solving.<n>Current benchmarks for measuring the intelligence of MLLMs suffer from limited scale, narrow coverage, and unstructured knowledge.<n>We introduce MDK12-Bench, a large-scale multidisciplinary benchmark built from real-world K-12 exams spanning six disciplines.
  arXiv  Detail & Related papers  (2025-08-09T06:21:10Z)
• 3LM: Bridging Arabic, STEM, and Code through Benchmarking [0.7227323884094952]
  3LM is a suite of three benchmarks designed specifically for Arabic.<n>The first is a set of STEM-related question-answer pairs, naturally sourced from Arabic textbooks and educational worksheets.<n>The second consists of synthetically generated STEM questions, created using the same sources.<n>The third benchmark focuses on code generation, built through a careful translation of two widely used code benchmarks.
  arXiv  Detail & Related papers  (2025-07-21T17:58:27Z)
• BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset [83.61028277413543]
  We introduce BMMR, a large-scale bilingual, multimodal, multi-disciplinary reasoning dataset for the community to develop and evaluate large multimodal models (LMMs)<n> BMMR comprises 110k college-level questions spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice, fill-in-the-blank, and open-ended QA-and sourced from both print and digital media such as books, exams, and quizzes.<n>The dataset is organized into two parts: BMMR-Eval that comprises 20,458 high-quality instances to comprehensively assess LMMs' knowledge and reasoning across multiple disciplines in both Chinese and
  arXiv  Detail & Related papers  (2025-07-04T11:20:09Z)
• TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish [54.51310112013655]
  We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models.
  arXiv  Detail & Related papers  (2024-07-17T08:28:55Z)
• FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models [64.11333762954283]
  This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities.
  arXiv  Detail & Related papers  (2024-04-29T01:49:07Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• Building Domain-Specific LLMs Faithful To The Islamic Worldview: Mirage or Technical Possibility? [0.0]
  Large Language Models (LLMs) have demonstrated remarkable performance across numerous natural language understanding use cases. In the context of Islam and its representation, accurate and factual representation of its beliefs and teachings rooted in the Quran and Sunnah is key. This work focuses on the challenge of building domain-specific LLMs faithful to the Islamic worldview.
  arXiv  Detail & Related papers  (2023-12-11T18:59:09Z)
• M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models [76.88692952308084]
  M3Exam is a benchmark for evaluating large language models (LLMs) in a multilingual, multimodal, and multilevel context. M3Exam contains 12,317 questions in 9 diverse languages with three educational levels. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text.
  arXiv  Detail & Related papers  (2023-06-08T13:21:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.