When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

• URL: http://arxiv.org/abs/2607.24077v1
• Date: Mon, 27 Jul 2026 07:19:00 GMT
• Title: When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents
• Abstract summary: Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems.<n>In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset.
• Score: 6.6457932580691015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optical Character Recognition (OCR) is a key component in the digitization of historical archives. Recently, Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems, achieving state-of-the-art performance on standard benchmarks. However, their suitability for archival transcription remains insufficiently understood. In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset, a challenging collection of Uruguayan dictatorship-era documents derived from microfilm scans. While VLMs consistently outperform traditional methods in terms of Character Error Rate (CER) and Word Error Rate (WER), we show that these improvements hide a more complex picture. Through a detailed qualitative analysis, we uncover systematic failure modes that are invisible to standard metrics, including orthographic normalization, spurious content generation, and semantic substitutions that preserve fluency while altering meaning. Errors affecting named entities are particularly critical, as they can introduce substantial semantic distortions with minimal impact on CER and WER. These findings reveal a critical gap between quantitative OCR performance and transcription fidelity in real-world archival settings, and highlight the need for evaluation frameworks that go beyond character-level accuracy to capture the semantic reliability of generated transcriptions.

Related papers

• TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents [50.64307290680222]
  TongGuOCR is a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents.<n>TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric.
  arXiv  Detail & Related papers  (2026-08-08T04:50:00Z)
• Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions [16.43811675687955]
  Vision-Language Models (VLMs) for optical character recognition can generate plausible but visually unsupported text.<n>We show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions.<n>Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents.
  arXiv  Detail & Related papers  (2026-05-26T22:57:01Z)
• Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation [98.79811866787263]
  We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes.<n>We identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision.<n>By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity.
  arXiv  Detail & Related papers  (2026-05-19T15:25:13Z)
• When Good OCR Is Not Enough: Benchmarking OCR Robustness for Retrieval-Augmented Generation [2.8437065210971872]
  Industrial Retrieval-Augmented Generation (RAG) systems depend on optical character recognition (OCR) to transform visual documents into text.<n>Existing OCR benchmarks rely on character-level metrics, which inadequately measure downstream RAG effectiveness under real-world conditions.<n>We introduce an OCR benchmark for industrial RAG systems covering 11 challenging document types.
  arXiv  Detail & Related papers  (2026-04-29T09:42:42Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model [0.07874708385247352]
  OCR of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography.<n>We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts.<n>TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation.
  arXiv  Detail & Related papers  (2026-02-16T07:17:52Z)
• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation [57.97548022208733]
  We show that seemingly superficial choices in key-value extraction can induce shifts in accuracy and stability.<n>We introduce Contextual Normalization, a strategy that adaptively standardizes context representations before generation.
  arXiv  Detail & Related papers  (2025-10-15T06:28:25Z)
• Layout-Aware OCR for Black Digital Archives with Unsupervised Evaluation [0.0]
  We present a layout-aware OCR pipeline tailored for Black newspaper archives.<n>Our approach integrates synthetic layout generation, model pretraining on augmented data, and a fusion of state-of-the-art You Only Look Once (YOLO) detectors.<n>Our results highlight the importance of respecting cultural layout logic in AI-driven document understanding.
  arXiv  Detail & Related papers  (2025-09-16T16:43:34Z)
• Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval [38.569818461453394]
  Retrieval-Augmented Generation (RAG) is a technique for grounding responses in external documents.<n>Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text.<n>Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR.
  arXiv  Detail & Related papers  (2025-05-08T21:54:02Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• Oracle Character Recognition using Unsupervised Discriminative Consistency Network [65.64172835624206]
  We propose a novel unsupervised domain adaptation method for oracle character recognition (OrCR) We leverage pseudo-labeling to incorporate the semantic information into adaptation and constrain augmentation consistency. Our approach achieves state-of-the-art result on Oracle-241 dataset and substantially outperforms the recently proposed structure-texture separation network by 15.1%.
  arXiv  Detail & Related papers  (2023-12-11T02:52:27Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.