A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents

• URL: http://arxiv.org/abs/2603.25761v1
• Date: Thu, 26 Mar 2026 02:52:28 GMT
• Title: A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents
• Abstract summary: This study examines how OCR and document understanding systems are evaluated, with particular attention to Black historical newspapers.<n>We found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation benchmarks.<n>To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optical character recognition (OCR) and document understanding systems increasingly rely on large vision and vision-language models, yet evaluation remains centered on modern, Western, and institutional documents. This emphasis masks system behavior in historical and marginalized archives, where layout, typography, and material degradation shape interpretation. This study examines how OCR and document understanding systems are evaluated, with particular attention to Black historical newspapers. We review OCR and document understanding papers, as well as benchmark datasets, which are published between 2006 and 2025 using the PRISMA framework. We look into how the studies report training data, benchmark design, and evaluation metrics for vision transformer and multimodal OCR systems. During the review, we found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation benchmarks. Most evaluations emphasize character accuracy and task success on modern layouts. They rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text. To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm. We propose that these gaps occur due to organizational (meso) and institutional (macro) behaviors and structure, shaped by benchmark incentives and data governance decisions.

Related papers

• When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents [6.6457932580691015]
  Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems.<n>In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset.
  arXiv  Detail & Related papers  (2026-07-27T07:19:00Z)
• Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters [31.3410141149246]
  We introduce Chronicles-OCR, the first comprehensive benchmark specifically designed to evaluate the cross-temporal visual perception capabilities of Vision Large Language Models.<n>The dataset comprises 2,800 strictly balanced images encompassing highly diverse physical media, ranging from tortoise shells to paper-based calligraphy.<n>Chronicles-OCR formulates four rigorous quantitative tasks: cross-period character spotting, fine-grained archaic character recognition via visual referring, ancient text parsing, and script classification.
  arXiv  Detail & Related papers  (2026-05-12T11:14:25Z)
• Structure-Aware Text Recognition for Ancient Greek Critical Editions [16.43811675687955]
  This paper investigates structure-aware text recognition for Ancient Greek critical editions.<n>We introduce a large-scale synthetic corpus of 185,000 page images generated from TEI/XML sources with controlled typographic and layout variation.<n>We evaluate three state-of-the-art visual language models under both zero-shot and fine-tuning regimes.
  arXiv  Detail & Related papers  (2026-03-03T09:42:43Z)
• ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review [48.60540055009675]
  ScholarPeer is a search-enabled multi-agent framework designed to emulate the cognitive processes of a senior researcher.<n>We evaluate ScholarPeer on DeepReview-13K and the results demonstrate that ScholarPeer achieves significant win-rates against state-of-the-art approaches in side-by-side evaluations.
  arXiv  Detail & Related papers  (2026-01-30T06:54:55Z)
• Layout-Aware OCR for Black Digital Archives with Unsupervised Evaluation [0.0]
  We present a layout-aware OCR pipeline tailored for Black newspaper archives.<n>Our approach integrates synthetic layout generation, model pretraining on augmented data, and a fusion of state-of-the-art You Only Look Once (YOLO) detectors.<n>Our results highlight the importance of respecting cultural layout logic in AI-driven document understanding.
  arXiv  Detail & Related papers  (2025-09-16T16:43:34Z)
• DREAM: Document Reconstruction via End-to-end Autoregressive Model [53.51754520966657]
  We present an innovative autoregressive model specifically designed for document reconstruction, referred to as Document Reconstruction via End-to-end Autoregressive Model (DREAM)<n>We establish a standardized definition of the document reconstruction task, and introduce a novel Document Similarity Metric (DSM) and DocRec1K dataset for assessing the performance of the task.
  arXiv  Detail & Related papers  (2025-07-08T09:24:07Z)
• Structured Attention Matters to Multimodal LLMs in Document Understanding [52.37530640460363]
  We investigate how input format influences document comprehension performance.<n>We discover that raw OCR text often impairs rather than improves MLLMs' performance.<n>We propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm.
  arXiv  Detail & Related papers  (2025-06-19T07:16:18Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Measuring Intersectional Biases in Historical Documents [37.03904311548859]
  We investigate the continuities and transformations of bias in historical newspapers published in the Caribbean during the colonial era (18th to 19th centuries) Our analyses are performed along the axes of gender, race, and their intersection. We find that there is a trade-off between the stability of the word embeddings and their compatibility with the historical dataset.
  arXiv  Detail & Related papers  (2023-05-21T07:10:31Z)
• Hierarchical Catalogue Generation for Literature Review: A Benchmark [36.22298354302282]
  We construct a novel English Hierarchical Catalogues of Literature Reviews dataset with 7.6k literature review catalogues and 389k reference papers. To accurately assess the model performance, we design two evaluation metrics for informativeness and similarity to ground truth from semantics and structure.
  arXiv  Detail & Related papers  (2023-04-07T07:13:35Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter [63.5550818034739]
  This paper presents a framework to evaluate state-of-the-art contributions to self-supervised monocular depth estimation. It includes pretraining, backbone, architectural design choices and loss functions. We re-implement, validate and re-evaluate 16 state-of-the-art contributions and introduce a new dataset.
  arXiv  Detail & Related papers  (2022-08-02T14:38:53Z)
• Robust Text Line Detection in Historical Documents: Learning and Evaluation Methods [1.9938405188113029]
  We present a study conducted using three state-of-the-art systems Doc-UFCN, dhSegment and ARU-Net. We show that it is possible to build generic models trained on a wide variety of historical document datasets that can correctly segment diverse unseen pages.
  arXiv  Detail & Related papers  (2022-03-23T11:56:25Z)
• Digital Editions as Distant Supervision for Layout Analysis of Printed Books [76.29918490722902]
  We describe methods for exploiting this semantic markup as distant supervision for training and evaluating layout analysis models. In experiments with several model architectures on the half-million pages of the Deutsches Textarchiv (DTA), we find a high correlation of these region-level evaluation methods with pixel-level and word-level metrics. We discuss the possibilities for improving accuracy with self-training and the ability of models trained on the DTA to generalize to other historical printed books.
  arXiv  Detail & Related papers  (2021-12-23T16:51:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.