Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models

• URL: http://arxiv.org/abs/2502.14901v1
• Date: Tue, 18 Feb 2025 11:10:48 GMT
• Title: Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models
• Abstract summary: Digitally archived journalism of Oscar Wilde's 19th century often has no or poor quality Optical Character Recognition (OCR)<n>This paper helps address the issue by performing OCR on "The Nineteenth Century Serials Edition" (NCSE), an 84k-page collection of 19th-century English newspapers and periodicals.<n>The resulting NCSE v2.0 dataset features improved article identification, high-quality OCR, and text classified into four types and seventeen topics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Oscar Wilde said, "The difference between literature and journalism is that journalism is unreadable, and literature is not read." Unfortunately, The digitally archived journalism of Oscar Wilde's 19th century often has no or poor quality Optical Character Recognition (OCR), reducing the accessibility of these archives and making them unreadable both figuratively and literally. This paper helps address the issue by performing OCR on "The Nineteenth Century Serials Edition" (NCSE), an 84k-page collection of 19th-century English newspapers and periodicals, using Pixtral 12B, a pre-trained image-to-text language model. The OCR capability of Pixtral was compared to 4 other OCR approaches, achieving a median character error rate of 1%, 5x lower than the next best model. The resulting NCSE v2.0 dataset features improved article identification, high-quality OCR, and text classified into four types and seventeen topics. The dataset contains 1.4 million entries, and 321 million words. Example use cases demonstrate analysis of topic similarity, readability, and event tracking. NCSE v2.0 is freely available to encourage historical and sociological research. As a result, 21st-century readers can now share Oscar Wilde's disappointment with 19th-century journalistic standards, reading the unreadable from the comfort of their own computers.

Related papers

• Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study [0.0]
  OCR systems report strong results on English and Chinese document benchmarks, yet their behaviour on Indic scripts is largely uncharacterised.<n>We benchmark ten systems on Devanagari (Hindi): classical EasyOCR; open VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B, olmOCR-7B); specialised OCR-VLMs (DeepSeek-OCR, Unlimited-OCR); and frontier closed models (Gemini 2.5 Flash, Claude Opus 4.7, GPT-5.5, Mistral OCR)<n>We release the benchmark, code, and models.
  arXiv  Detail & Related papers  (2026-06-28T05:46:05Z)
• The Devil is in the Details -- From OCR for Old Church Slavonic to Purely Visual Stemma Reconstruction [0.0]
  Investigation starts comparing a range of OCR-systems, from classical over machine learning to LLMs, for roughly 6,000 characters of late handwritten church slavonic manuscripts from the 18th century.<n>Experiments suggest, church slavonic CER for basic letters may reach as low as 2-3% but elaborated diacritics could still present a problem.<n>How well OCR can prime stemmatology as a downstream task is the entry point to the second part of the article.
  arXiv  Detail & Related papers  (2026-04-13T16:58:57Z)
• ICDAR 2025 Competition on End-to-End Document Image Machine Translation Towards Complex Layouts [47.61983500021015]
  Document Image Machine Translation (DIMT) seeks to translate text embedded in document images from one language to another.<n>The DIMT 2025 Challenge advances research on end-to-end document image translation.<n>The competition features two tracks, OCR-free and OCR-based, each with two subtasks for small (less than 1B parameters) and large (greater than 1B parameters) models.
  arXiv  Detail & Related papers  (2026-03-10T09:04:38Z)
• Post-OCR Text Correction for Bulgarian Historical Documents [31.072768715994318]
  We create the first benchmark dataset for evaluating the OCR text correction for historical Bulgarian documents written in the first standardized Bulgarian orthography: the Drinov orthography from the 19th century. We then use state-of-the-art LLMs and encoder-decoder framework which we augment with diagonal attention loss and copy and coverage mechanisms to improve the post-OCR text correction. The proposed method reduces the errors introduced during recognition and improves the quality of the documents by 25%, which is an increase of 16% compared to the state-of-the-art on the ICDAR 2019
  arXiv  Detail & Related papers  (2024-08-31T19:27:46Z)
• Making Old Kurdish Publications Processable by Augmenting Available Optical Character Recognition Engines [1.174020933567308]
  Kurdish libraries have many historical publications that were printed back in the early days when printing devices were brought to Kurdistan. Current Optical Character Recognition (OCR) systems are unable to extract text from historical documents as they have many issues. In this study, we adopt an open-source OCR framework by Google, Tesseract version 5.0, that has been used to extract text for various languages.
  arXiv  Detail & Related papers  (2024-04-09T08:08:03Z)
• LOCR: Location-Guided Transformer for Optical Character Recognition [55.195165959662795]
  We propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.
  arXiv  Detail & Related papers  (2024-03-04T15:34:12Z)
• Improving Multimodal Datasets with Image Captioning [65.74736570293622]
  We study how generated captions can increase the utility of web-scraped datapoints with nondescript text. Our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text.
  arXiv  Detail & Related papers  (2023-07-19T17:47:12Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)
• A Benchmark and Dataset for Post-OCR text correction in Sanskrit [23.45279030301887]
  Sanskrit is a classical language with about 30 million extant manuscripts fit for digitisation. We release a post-OCR text correction dataset containing around 218,000 sentences, with 1.5 million words, from 30 different books.
  arXiv  Detail & Related papers  (2022-11-15T08:32:18Z)
• PART: Pre-trained Authorship Representation Transformer [52.623051272843426]
  Authors writing documents imprint identifying information within their texts.<n>Previous works use hand-crafted features or classification tasks to train their authorship models.<n>We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection [83.3580786484122]
  We find that newspapers from larger schools, located in wealthier, educated, and urban ZIP codes are more likely to be classified as high quality. We argue that privileging any corpus as high quality entails a language ideology.
  arXiv  Detail & Related papers  (2022-01-25T17:20:04Z)
• Optical Character Recognition of 19th Century Classical Commentaries: the Current State of Affairs [0.0]
  We evaluate the performances of two pipelines suitable for the OCR of historical classical commentaries. Our results show that Kraken + Ciaconna reaches a substantially lower character error rate (CER) than Tesseract/OCR-D on commentary sections with high density of polytonic Greek text. As part of this paper, we also release GT4HistComment, a small dataset with OCR ground truth for 19th classical commentaries and Pogretra, a large collection of training data and pre-trained models for a wide variety of ancient Greek typefaces.
  arXiv  Detail & Related papers  (2021-10-13T16:01:16Z)
• OCR Post Correction for Endangered Language Texts [113.8242302688894]
  We create a benchmark dataset of transcriptions for scanned books in three critically endangered languages. We present a systematic analysis of how general-purpose OCR tools are not robust to the data-scarce setting. We develop an OCR post-correction method tailored to ease training in this data-scarce setting.
  arXiv  Detail & Related papers  (2020-11-10T21:21:08Z)
• The Newspaper Navigator Dataset: Extracting And Analyzing Visual Content from 16 Million Historic Newspaper Pages in Chronicling America [10.446473806802578]
  We introduce a visual content recognition model trained on bounding box annotations of photographs, illustrations, maps, comics, and editorial cartoons. We describe our pipeline that utilizes this deep learning model to extract 7 classes of visual content. We report the results of running the pipeline on 16.3 million pages from the Chronicling America corpus.
  arXiv  Detail & Related papers  (2020-05-04T15:51:13Z)
• Deconfounded Image Captioning: A Causal Retrospect [89.0694058049884]
  We present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0. DICv1.0 is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split
  arXiv  Detail & Related papers  (2020-03-09T04:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.