Pretraining Language Models on Historical Text

• URL: http://arxiv.org/abs/2606.02991v1
• Date: Tue, 02 Jun 2026 00:59:06 GMT
• Title: Pretraining Language Models on Historical Text
• Abstract summary: We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913.<n>We construct TypewriterCorpus, a 54B-token historical corpus with extensive data cleaning and leakage mitigation procedures.<n>We also introduce lexically grounded instructing tuning, a post-training framework that constraints responses to remain directly grounded in historical source documents.
• Score: 36.643370581003055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913. Developing History LMs requires addressing challenges in data quality and availability, preventing temporal leakage, designing temporally consistent post-training pipelines, and constructing reliable evaluations. To address these issues, we construct TypewriterCorpus, a 54B-token historical corpus collected from diverse archival and linguistically annotated sources with extensive data cleaning and leakage mitigation procedures. Furthermore, we introduce lexically grounded instructing tuning, a post-training framework that constraints responses to remain directly grounded in historical source documents. Using this framework we construct two historical instruction tuning datasets: History-LIMA and History-SelfInstruct. To evaluate capability and temporal consistency, we introduce History-Event, a benchmark suite for evaluating competence, temporal grounding and data leakage. We release TypewriterLM and all associated resources to support future research on historical language models.

Related papers

• Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models [1.0026496861838445]
  We introduce a novel framework for historical document restoration that leverages large language models with retrieval-augmented generation (RAG)<n>Our approach significantly outperforms baselines, achieving substantial gains in restoring both general characters and named entities.<n> comprehensive evaluations including expert assessments confirm that ARI serves as a practical tool for domain experts, promising to accelerate the analysis of historical records.
  arXiv  Detail & Related papers  (2026-07-24T03:11:25Z)
• Preserving Historical Truth: Detecting Historical Revisionism in Large Language Models [66.75310318710073]
  We introduce textttHistoricalMisinfo, a curated dataset of $500$ contested events from $45$ countries.<n>To approximate real-world usage, we instantiate each event in $11$ prompt scenarios that reflect common communication settings.
  arXiv  Detail & Related papers  (2026-02-19T15:05:10Z)
• Named Entity Recognition of Historical Text via Large Language Model [1.5469452301122175]
  Named Entity Recognition (NER) plays a crucial role in extracting information from unstructured text.<n>Traditionally, NER is addressed using supervised machine learning approaches, which require large amounts of annotated training data.<n>In this study, we explore the feasibility of applying LLMs to NER in historical documents using zero-shot and few-shot prompting strategies.<n>Our experiments, conducted on the HIPE-2022 dataset, show that LLMs can achieve reasonably strong performance on NER tasks in this setting.
  arXiv  Detail & Related papers  (2025-08-25T14:52:11Z)
• LLM Agents for Interactive Exploration of Historical Cadastre Data: Framework and Application to Venice [2.03659124799413]
  Cadastral data reveal key information about the historical organization of cities but are often non-standardized due to diverse formats and human annotations.<n>We explore as a case study Venice's urban history during the critical period from 1740 to 1808.<n>This era's complex cadastral data, marked by its volume and lack of uniform structure, presents unique challenges that our approach adeptly navigates.
  arXiv  Detail & Related papers  (2025-05-22T08:45:15Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model.<n>Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones.<n>Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models [0.0]
  This paper introduces foundational resources and models for natural language processing (NLP) of historical Turkish.<n>We present the first named entity recognition (NER) dataset, HisTR and the first Universal Dependencies treebank, OTA-BOUN for a historical form of the Turkish language.<n>We also introduce Ottoman Text Corpus (OTC), a clean corpus of transliterated historical Turkish texts.
  arXiv  Detail & Related papers  (2025-01-08T20:29:00Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System [59.78425104243993]
  We propose History-Aware Hierarchical Transformer (HAHT) for multi-session open-domain dialogue. HAHT maintains a long-term memory of history conversations and utilizes history information to understand current conversation context. Experimental results on a large-scale Multi-Session Conversation dataset suggest that the proposed HAHT model consistently outperforms baseline models.
  arXiv  Detail & Related papers  (2023-02-02T06:54:33Z)
• hmBERT: Historical Multilingual Language Models for Named Entity Recognition [0.6226609932118123]
  We tackle NER for identifying persons, locations, and organizations in historical texts. In this work, we tackle NER for historical German, English, French, Swedish, and Finnish by training large historical language models.
  arXiv  Detail & Related papers  (2022-05-31T07:30:33Z)
• From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French [57.886210204774834]
  We present our efforts to develop NLP tools for Early Modern French (historical French from the 16$textth$ to the 18$textth$ centuries). We present the $textFreEM_textmax$ corpus of Early Modern French and D'AlemBERT, a RoBERTa-based language model trained on $textFreEM_textmax$.
  arXiv  Detail & Related papers  (2022-02-18T22:17:22Z)
• How Context Affects Language Models' Factual Predictions [134.29166998377187]
  We integrate information from a retrieval system with a pre-trained language model in a purely unsupervised way. We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline.
  arXiv  Detail & Related papers  (2020-05-10T09:28:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.