Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

• URL: http://arxiv.org/abs/2604.24690v1
• Date: Mon, 27 Apr 2026 16:50:40 GMT
• Title: Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination
• Abstract summary: ProHist-Bench is a novel benchmark anchored in the Chinese Imperial Examination (Keju) system.<n>It features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics.<n>We reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions.
• Score: 11.650720838376634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professional-level historical reasoning remains underexplored. Existing benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills, such as evidentiary reasoning,that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel benchmark anchored in the Chinese Imperial Examination (Keju) system, a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench.

Related papers

• Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective [53.594353527056775]
  We propose Chinese Commonsense Multi-hop Reasoning ( CCMOR) to evaluate Large Language Models (LLMs)<n> CCMOR is designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning.<n>We implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions.
  arXiv  Detail & Related papers  (2025-10-09T20:29:00Z)
• On Path to Multimodal Historical Reasoning: HistBench and HistAgent [68.02249599465337]
  HistBench is a new benchmark of 414 high-quality questions designed to evaluate AI's capacity for historical reasoning.<n>Tasks span a wide range of historical problems-from factual retrieval based on primary sources to interpretive analysis of manuscripts and images.<n>We present HistAgent, a history-specific agent equipped with carefully designed tools for OCR, translation, archival search, and image understanding in History.
  arXiv  Detail & Related papers  (2025-05-26T17:22:20Z)
• Kongzi: A Historical Large Language Model with Fact Enhancement [4.687722574822698]
  Kongzi is a large language model specifically designed for historical analysis.<n>Through the integration of curated, high-quality historical data and a novel fact-reinforcement learning strategy, Kongzi demonstrates strong factual alignment and sophisticated reasoning depth.
  arXiv  Detail & Related papers  (2025-04-13T09:01:05Z)
• Thus Spake Long-Context Large Language Model [70.49178031298953]
  Long context is an important topic in Natural Language Processing (NLP)<n>It offers immense opportunities for Large Language Models (LLMs) giving LLMs the lifelong learning potential akin to humans.<n>In the past two years, the context length of LLMs has achieved a breakthrough extension to millions of tokens.<n>Research on long-context LLMs has expanded from length extrapolation to a comprehensive focus on architecture, infrastructure, training, and evaluation technologies.
  arXiv  Detail & Related papers  (2025-02-24T13:19:33Z)
• HistoLens: An LLM-Powered Framework for Multi-Layered Analysis of Historical Texts -- A Case Application of Yantie Lun [0.43512163406552007]
  HistoLens is a multi-layered analysis framework for historical texts based on Large Language Models (LLMs) HistoLens integrates NLP technology, including named entity recognition, knowledge graph construction, and geographic information visualization. This paper showcases how HistoLens explores Western Han culture in "Yantie Lun" through multi-dimensional, visual, and quantitative methods.
  arXiv  Detail & Related papers  (2024-11-15T06:21:13Z)
• MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs [61.74749961334557]
  MathHay is an automated benchmark designed to assess the long-context mathematical reasoning capabilities of LLMs. We conduct extensive experiments on MathHay to assess the long-context mathematical reasoning abilities of eight top-performing models.
  arXiv  Detail & Related papers  (2024-10-07T02:30:07Z)
• LongIns: A Challenging Long-context Instruction-based Exam for LLMs [20.8470269815788]
  Long-context capabilities of large language models (LLMs) have been a hot topic in recent years.<n>We propose the LongIns benchmark dataset, a challenging long-context instruction-based exam for LLMs.
  arXiv  Detail & Related papers  (2024-06-25T14:31:26Z)
• Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks [76.43527940649939]
  We introduce Ada-LEval, a benchmark for evaluating the long-context understanding of large language models (LLMs) Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable a more reliable evaluation of LLMs' long context capabilities. We evaluate 4 state-of-the-art closed-source API models and 6 open-source models with Ada-LEval.
  arXiv  Detail & Related papers  (2024-04-09T17:30:48Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [109.8527403904657]
  We show that large language models (LLMs) possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We propose a simple method to dynamically utilize supporting documents with our judgement strategy.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.