Historical Ink: 19th Century Latin American Spanish Newspaper Corpus with LLM OCR Correction

• URL: http://arxiv.org/abs/2407.12838v2
• Date: Fri, 4 Oct 2024 18:43:47 GMT
• Title: Historical Ink: 19th Century Latin American Spanish Newspaper Corpus with LLM OCR Correction
• Abstract summary: It introduces a novel dataset of 19th-century Latin American newspaper texts, addressing a critical gap in specialized corpora for historical and linguistic analysis in this region. It develops a flexible framework that utilizes a Large Language Model for OCR error correction and linguistic surface form detection in digitized corpora.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents two significant contributions: First, it introduces a novel dataset of 19th-century Latin American newspaper texts, addressing a critical gap in specialized corpora for historical and linguistic analysis in this region. Second, it develops a flexible framework that utilizes a Large Language Model for OCR error correction and linguistic surface form detection in digitized corpora. This semi-automated framework is adaptable to various contexts and datasets and is applied to the newly created dataset.

Related papers

• On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation [79.18001266925245]
  LingT2I is a benchmark covering 10 widely used languages with 33K prompts.<n>We conduct a comprehensive cross-lingual analysis, uncovering linguistic inequality and language-dependent trade-offs.
  arXiv  Detail & Related papers  (2026-08-11T14:49:23Z)
• Diffusion-Based Ukrainian Handwritten Text Generation with Cross-Domain Style Transfer [0.0]
  We construct a Ukrainian handwritten word dataset of 126,177 images from 308 writers using connected-component segmentation, quality filtering, and targeted oversampling of underrepresented Ukrainian characters.<n>We retrain a MobileNetV2 triplet-loss style encoder with a CANINE-conditioned latent diffusion U-Net, testing direct transfer from Latin to Cyrillic.<n>We evaluate cross-domain style transfer in three settings: cross-lingual transfer from English samples, zero-shot transfer to an early 20th-century Ukrainian manuscript, and few-shot imitation of contemporary writers.
  arXiv  Detail & Related papers  (2026-05-26T15:28:21Z)
• The Patrologia Graeca Corpus: OCR, Annotation, and Open Release of Noisy Nineteenth-Century Polytonic Greek Editions [0.0]
  We present the Patrologia Graeca Corpus, the first large-scale open OCR and linguistic resource for nineteenthcentury editions of Ancient Greek.<n>The collection covers the remaining undigitized volumes of the Patrologia Graeca (PG), printed in complex bilingual (Greek-Latin) layouts and characterized by highly degraded polytonic Greek typography.<n>We achieve a character error rate (CER) of 1.05% and a word error rate (WER) of 4.69%, largely outperforming existing OCR systems for polytonic Greek.<n>The resulting corpus contains around six million lemmatized and part-of-speech tagged tokens, aligned with full
  arXiv  Detail & Related papers  (2026-03-10T10:21:54Z)
• Historical Ink: Exploring Large Language Models for Irony Detection in 19th-Century Spanish [0.06554326244334868]
  This study explores the use of large language models (LLMs) to enhance datasets and improve irony detection in 19th-century Latin American newspapers. Two strategies were employed to evaluate the efficacy of BERT and GPT-4o models in capturing the subtle nuances nature of irony.
  arXiv  Detail & Related papers  (2025-03-28T16:33:24Z)
• Reference-Based Post-OCR Processing with LLM for Diacritic Languages [0.0]
  We propose a method utilizing available content-focused ebooks as a reference base to correct imperfect OCR-generated text. This technique generates high-precision pseudo-page-to-page labels for diacritic languages. The pipeline eliminates various types of noise from aged documents and addresses issues such as missing characters, words, and disordered sequences.
  arXiv  Detail & Related papers  (2024-10-17T08:05:02Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Post-OCR Text Correction for Bulgarian Historical Documents [31.072768715994318]
  We create the first benchmark dataset for evaluating the OCR text correction for historical Bulgarian documents written in the first standardized Bulgarian orthography: the Drinov orthography from the 19th century. We then use state-of-the-art LLMs and encoder-decoder framework which we augment with diagonal attention loss and copy and coverage mechanisms to improve the post-OCR text correction. The proposed method reduces the errors introduced during recognition and improves the quality of the documents by 25%, which is an increase of 16% compared to the state-of-the-art on the ICDAR 2019
  arXiv  Detail & Related papers  (2024-08-31T19:27:46Z)
• Spanish TrOCR: Leveraging Transfer Learning for Language Adaptation [0.0]
  This study explores the transfer learning capabilities of the TrOCR architecture to Spanish. We integrate an English TrOCR encoder with a language specific decoder and train the model on this specific language. Fine-tuning the English TrOCR on Spanish yields superior recognition than the language specific decoder for a fixed dataset size.
  arXiv  Detail & Related papers  (2024-07-09T15:31:41Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers [7.161822501147275]
  This study develops a novel, deep learning pipeline for extracting full article texts from newspaper images. It applies it to the nearly 20 million scans in Library of Congress's public domain Chronicling America collection. The pipeline includes layout detection, legibility classification, custom OCR, and association of article texts spanning multiple bounding boxes.
  arXiv  Detail & Related papers  (2023-08-24T00:24:42Z)
• Transformer-based Model for Word Level Language Identification in Code-mixed Kannada-English Texts [55.41644538483948]
  We propose the use of a Transformer based model for word-level language identification in code-mixed Kannada English texts. The proposed model on the CoLI-Kenglish dataset achieves a weighted F1-score of 0.84 and a macro F1-score of 0.61.
  arXiv  Detail & Related papers  (2022-11-26T02:39:19Z)
• Models and Datasets for Cross-Lingual Summarisation [78.56238251185214]
  We present a cross-lingual summarisation corpus with long documents in a source language associated with multi-sentence summaries in a target language. The corpus covers twelve language pairs and directions for four European languages, namely Czech, English, French and German. We derive cross-lingual document-summary instances from Wikipedia by combining lead paragraphs and articles' bodies from language aligned Wikipedia titles.
  arXiv  Detail & Related papers  (2022-02-19T11:55:40Z)
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN [63.79300884115027]
  Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? We introduce RAVEN, a suite of analyses for assessing the novelty of generated text.
  arXiv  Detail & Related papers  (2021-11-18T04:07:09Z)
• A Framework for Neural Topic Modeling of Text Corpora [6.340447411058068]
  We introduce FAME, an open-source framework enabling an efficient mechanism of extracting and incorporating textual features. To demonstrate the effectiveness of this library, we conducted experiments on the well-known News-Group dataset.
  arXiv  Detail & Related papers  (2021-08-19T23:32:38Z)
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation [100.09099800591822]
  XGLUE is a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models. XGLUE provides 11 diversified tasks that cover both natural language understanding and generation scenarios.
  arXiv  Detail & Related papers  (2020-04-03T07:03:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.