Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

• URL: http://arxiv.org/abs/2605.27750v1
• Date: Tue, 26 May 2026 22:57:01 GMT
• Title: Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
• Abstract summary: Vision-Language Models (VLMs) for optical character recognition can generate plausible but visually unsupported text.<n>We show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions.<n>Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents.
• Score: 16.43811675687955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually unsupported text, suggesting reliance on language priors. Comparing open-weight VLMs with traditional OCR baselines on low-resource Ancient Greek critical editions, we show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions where traditional engines produce local recognition noise. To analyze visual evidence during decoding, we introduce controlled image perturbations and token-level grounding measures based on conditional versus image-free decoding distributions. Under character-level perturbations, VLMs diverge sharply from the perturbed ground truth while traditional OCR remains comparatively faithful; however, token-level analysis shows that prior reliance is model-specific: in an OCR-specialist model, fluent lexical errors are produced with little reliance on the image, whereas general-purpose VLMs remain conditioned on the visual input even when wrong. Decode-time interventions fail to reliably restore grounding, while post-OCR language-model correction improves several systems only by repairing text after generation. Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents and a broader set of models, showing that fluent output is not necessarily visually grounded and motivating interpretability-driven evaluation beyond aggregate accuracy.

Related papers

• When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study [9.108791469266345]
  We evaluate traditional OCR, general-purpose VLMs, Arabic-specialized VLMs, and OCR-conditioned VLM correction across eight Arabic text datasets.<n> OCR-prior recoverability helps when the OCR output remains visually and textually recoverable, providing anchors that the VLM can refine against the image.<n>Additional diagnostics show that Arabic VLM-OCR datasets is sensitive to diacritics, preprocessing, generation budget, and repetition loops.
  arXiv  Detail & Related papers  (2026-08-23T11:12:06Z)
• Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning [21.36422191204697]
  Recent studies mainly improve visual counter-commonsense reasoning by enhancing visual inputs.<n>We introduce TACT, a text-anchored post-training framework that debiases the shared language decoder without requiring any visual training data.<n>TACT substantially improves visual reasoning while preserving general capabilities, demonstrating effective text-to-vision cross-modal transfer.
  arXiv  Detail & Related papers  (2026-08-07T08:10:31Z)
• Can Vision-Language Models Reason about AI Edits in Images? [78.04125956307838]
  Vision-Language Models (VLMs) offer a promising alternative due to their strong visual understanding and reasoning capabilities.<n>We propose a GRPO-based training framework that utilizes simple accuracy and format rewards.<n>Our approach achieves competitive detection and localization performance compared to state-of-the-art image forgery detectors.
  arXiv  Detail & Related papers  (2026-07-30T16:23:40Z)
• When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents [6.6457932580691015]
  Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems.<n>In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset.
  arXiv  Detail & Related papers  (2026-07-27T07:19:00Z)
• Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models [14.752088383510788]
  Vision-language models (VLMs) are often deployed on text-only inputs, although they are trained with images.<n>We find that removing the vision modality causes large drops in accuracy and severe miscalibration, and the model does not behave like its original language backbone under text-only prompting.<n>We propose the Latent Imagination Module (LIM), a lightweight cross-attention module that predicts imagined latent embeddings from textual input and feeds them into a frozen VLM backbone without pixel-level image synthesis.
  arXiv  Detail & Related papers  (2026-04-03T10:03:02Z)
• Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model [0.07874708385247352]
  OCR of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography.<n>We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts.<n>TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation.
  arXiv  Detail & Related papers  (2026-02-16T07:17:52Z)
• Adapting Language Balance in Code-Switching Speech [60.296574524609575]
  Large foundational models still struggle against code-switching test cases.<n>We use differentiable surrogates to mitigate context bias during generation.<n>Experiments with Arabic and Chinese-English showed that the models are able to predict the switching places more correctly.
  arXiv  Detail & Related papers  (2025-10-21T15:23:55Z)
• Vision Language Models Are Not (Yet) Spelling Correctors [0.742779257315787]
  Spelling correction from visual input poses unique challenges for vision language models (VLMs)<n>We present ReViCo, the first benchmark that systematically evaluates VLMs on real-world visual spelling correction across Chinese and English.
  arXiv  Detail & Related papers  (2025-09-22T07:10:42Z)
• DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model [9.557159109747372]
  Large vision-language models (LVLMs) are prone to hallucinations--generating words that do not exist in input images.<n>We propose DianJin-OCR-R1, a reasoning-and-tool interleaved VLMs trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2025-08-18T03:28:57Z)
• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment [57.0121616203175]
  We propose FiSAO, a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment.<n>By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data.
  arXiv  Detail & Related papers  (2024-10-18T03:34:32Z)
• Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning [90.13978453378768]
  We introduce a comprehensive typology of factual errors in generated chart captions. A large-scale human annotation effort provides insight into the error patterns and frequencies in captions crafted by various chart captioning models. Our analysis reveals that even state-of-the-art models, including GPT-4V, frequently produce captions laced with factual inaccuracies.
  arXiv  Detail & Related papers  (2023-12-15T19:16:21Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)
• Lexically Aware Semi-Supervised Learning for OCR Post-Correction [90.54336622024299]
  Much of the existing linguistic data in many languages of the world is locked away in non-digitized books and documents. Previous work has demonstrated the utility of neural post-correction methods on recognition of less-well-resourced languages. We present a semi-supervised learning method that makes it possible to utilize raw images to improve performance.
  arXiv  Detail & Related papers  (2021-11-04T04:39:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.