Enriching Historical Records: An OCR and AI-Driven Approach for Database Integration

• URL: http://arxiv.org/abs/2512.23710v1
• Date: Mon, 17 Nov 2025 15:13:42 GMT
• Title: Enriching Historical Records: An OCR and AI-Driven Approach for Database Integration
• Abstract summary: This research digitizes and analyzes the Leidse hoogleraren en lectoren 1575-1815 books written between 1983 and 1985.<n>It addresses the central question: how can we design an automated pipeline that integrates OCR, interpretation, and database linking to harmonize data from historical document images with existing high-quality database records?<n>We applied OCR techniques, generative AI decoding constraints that structure data extraction, and database linkage methods to process type historical records into a digital format OCR achieved a Character Error Rate (CER) of 1.08 percent and a Word Error Rate (WER) of 5.06 percent, while extraction from OCR
• Score: 3.184882058033374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This research digitizes and analyzes the Leidse hoogleraren en lectoren 1575-1815 books written between 1983 and 1985, which contain biographic data about professors and curators of Leiden University. It addresses the central question: how can we design an automated pipeline that integrates OCR, LLM-based interpretation, and database linking to harmonize data from historical document images with existing high-quality database records? We applied OCR techniques, generative AI decoding constraints that structure data extraction, and database linkage methods to process typewritten historical records into a digital format. OCR achieved a Character Error Rate (CER) of 1.08 percent and a Word Error Rate (WER) of 5.06 percent, while JSON extraction from OCR text achieved an average accuracy of 63 percent and, based on annotated OCR, 65 percent. This indicates that generative AI somewhat corrects low OCR performance. Our record linkage algorithm linked annotated JSON files with 94% accuracy and OCR-derived JSON files with 81%. This study contributes to digital humanities research by offering an automated pipeline for interpreting digitized historical documents, addressing challenges like layout variability and terminology differences, and exploring the applicability and strength of an advanced generative AI model.

Related papers

• TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents [50.64307290680222]
  TongGuOCR is a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents.<n>TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric.
  arXiv  Detail & Related papers  (2026-08-08T04:50:00Z)
• Improving Access to Historical Archives with Real-time RAG-based Systems [16.569566038184625]
  We present an end-to-end archival processing and retrieval framework that integrates large language models.<n>Our system introduces two core components: (i) an OCR refinement module that improves text quality, and (ii) a semantic retrieval and cross-encoder reranking pipeline.
  arXiv  Detail & Related papers  (2026-07-03T15:53:12Z)
• Test-Time Strategies for More Efficient and Accurate Agentic RAG [58.44913384057518]
  Retrieval-Augmented Generation (RAG) systems face challenges with complex, multihop questions.<n>Such approaches can introduce inefficiencies, including repetitive retrieval of previously processed information.<n>We investigate test-time modifications to the Search-R1 pipeline to mitigate these identified shortcomings.
  arXiv  Detail & Related papers  (2026-03-12T19:18:59Z)
• SciCoQA: Quality Assurance for Scientific Paper--Code Alignment [53.70401063640645]
  We present SciCoQA, a dataset for detecting discrepancies between scientific publications and theirs.<n>Our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines.<n>The best performing model in our evaluation, GPT-5, can only detect 45.7% of real-world paper-code discrepancies.
  arXiv  Detail & Related papers  (2026-01-19T10:04:33Z)
• HySemRAG: A Hybrid Semantic Retrieval-Augmented Generation Framework for Automated Literature Synthesis and Methodological Gap Analysis [55.2480439325792]
  HySemRAG is a framework that combines Extract, Transform, Load (ETL) pipelines with Retrieval-Augmented Generation (RAG)<n>System addresses limitations in existing RAG architectures through a multi-layered approach.
  arXiv  Detail & Related papers  (2025-08-01T20:30:42Z)
• Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models [92.85086256871027]
  We propose REWIRE, REcycling the Web with guIded REwrite, to enrich low-quality documents so that they could become useful for training.<n>We demonstrate that about 82% of the mixed in texts come from transforming lower-quality documents that would otherwise be discarded.
  arXiv  Detail & Related papers  (2025-06-05T07:12:12Z)
• Reference-Based Post-OCR Processing with LLM for Precise Diacritic Text in Historical Document Recognition [1.6941039309214678]
  We propose a method utilizing available content-focused ebooks as a reference base to correct imperfect OCR-generated text.<n>This technique generates high-precision pseudo-page-to-page labels for diacritic languages.<n>The pipeline eliminates various types of noise from aged documents and addresses issues such as missing characters, words, and disordered sequences.
  arXiv  Detail & Related papers  (2024-10-17T08:05:02Z)
• CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models [0.0]
  This paper introduces Context Leveraging OCR Correction (CLOCR-C)<n>It uses the infilling and context-adaptive abilities of transformer-based language models (LMs) to improve OCR quality.<n>The study aims to determine if LMs can perform post-OCR correction, improve downstream NLP tasks, and the value of providing the socio-cultural context as part of the correction process.
  arXiv  Detail & Related papers  (2024-08-30T17:26:05Z)
• Making Old Kurdish Publications Processable by Augmenting Available Optical Character Recognition Engines [1.174020933567308]
  Kurdish libraries have many historical publications that were printed back in the early days when printing devices were brought to Kurdistan. Current Optical Character Recognition (OCR) systems are unable to extract text from historical documents as they have many issues. In this study, we adopt an open-source OCR framework by Google, Tesseract version 5.0, that has been used to extract text for various languages.
  arXiv  Detail & Related papers  (2024-04-09T08:08:03Z)
• LOCR: Location-Guided Transformer for Optical Character Recognition [55.195165959662795]
  We propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.
  arXiv  Detail & Related papers  (2024-03-04T15:34:12Z)
• How Does Generative Retrieval Scale to Millions of Passages? [68.98628807288972]
  We conduct the first empirical study of generative retrieval techniques across various corpus scales. We scale generative retrieval to millions of passages with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. While generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge.
  arXiv  Detail & Related papers  (2023-05-19T17:33:38Z)
• Lights, Camera, Action! A Framework to Improve NLP Accuracy over OCR documents [2.6201102730518606]
  We demonstrate an effective framework for mitigating OCR errors for any downstream NLP task. We first address the data scarcity problem for model training by constructing a document synthesis pipeline. For the benefit of the community, we have made the document synthesis pipeline available as an open-source project.
  arXiv  Detail & Related papers  (2021-08-06T00:32:54Z)
• On Addressing Practical Challenges for RNN-Transduce [72.72132048437751]
  We adapt a well-trained RNN-T model to a new domain without collecting the audio data. We obtain word-level confidence scores by utilizing several types of features calculated during decoding. The proposed time stamping method can get less than 50ms word timing difference on average.
  arXiv  Detail & Related papers  (2021-04-27T23:31:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.