Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models

• URL: http://arxiv.org/abs/2607.21617v1
• Date: Fri, 29 May 2026 20:33:35 GMT
• Title: Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
• Abstract summary: Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding.<n>We show they do not always act as faithful transcribers: when text is imperfect, they often rewrite it into a more plausible form.
• Score: 10.628090041273929
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding. In this paper, we show they do not always act as faithful transcribers: when text is imperfect, they often tend to rewrite it into a more plausible form - a behavior that clean-text OCR benchmarks cannot detect. We introduce FaithC4, a multilingual perturbation benchmark of 1,455 single-page documents (English, Chinese, Korean) with three perturbation families: scramble, random substitution, and visually similar substitution. We use the benchmark to evaluate 15 systems spanning general-purpose VLMs, OCR-specialized VLMs, and traditional OCR pipelines. These three categories differ in WER degradation under perturbation: general-purpose VLMs degrade by up to 4.5 points, OCR-specialized VLMs by 0.2-2 points, and traditional OCR by less than 0.6 points on English. Probing Qwen3-VL-4B layer-by-layer, we identify a consistent pattern: rewriting fires only when a perturbed word's final layer FFN representation stays close to the original encoding; when the representation diverges sufficiently, the model transcribes faithfully. Word length affects rewriting rate: short words (4-6 characters) are rewritten up to 10% of the time, with a sharp cutoff at 8 characters above which rewriting drops to 0%.

Related papers

• When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study [9.108791469266345]
  We evaluate traditional OCR, general-purpose VLMs, Arabic-specialized VLMs, and OCR-conditioned VLM correction across eight Arabic text datasets.<n> OCR-prior recoverability helps when the OCR output remains visually and textually recoverable, providing anchors that the VLM can refine against the image.<n>Additional diagnostics show that Arabic VLM-OCR datasets is sensitive to diacritics, preprocessing, generation budget, and repetition loops.
  arXiv  Detail & Related papers  (2026-08-23T11:12:06Z)
• TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents [50.64307290680222]
  TongGuOCR is a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents.<n>TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric.
  arXiv  Detail & Related papers  (2026-08-08T04:50:00Z)
• When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning [73.56343820592399]
  We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed.<n>Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average.<n>We present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view.
  arXiv  Detail & Related papers  (2026-08-05T11:46:15Z)
• BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models [0.0]
  We introduce BANGLAWILD, a benchmark of 2,535 Bengali scene text images.<n>We evaluate fifteen VLMs and three conventional OCR systems under three prompting strategies.<n>Our results reveal a persistent gap in which larger models within the same family do not outperform smaller ones.
  arXiv  Detail & Related papers  (2026-08-04T16:20:53Z)
• When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscripts [3.3982074915384874]
  We evaluate the full image-to-translation pipeline on medieval Latin manuscripts.<n> domain-specific Optical Character Recognition (OCR) models reduce character error rate by up to 4.3$times$ compared to general-purpose VLMs.<n>These findings offer both a new benchmark and practical guidance for deploying translation systems in low-resource historical settings.
  arXiv  Detail & Related papers  (2026-07-04T11:58:52Z)
• Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study [0.0]
  OCR systems report strong results on English and Chinese document benchmarks, yet their behaviour on Indic scripts is largely uncharacterised.<n>We benchmark ten systems on Devanagari (Hindi): classical EasyOCR; open VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B, olmOCR-7B); specialised OCR-VLMs (DeepSeek-OCR, Unlimited-OCR); and frontier closed models (Gemini 2.5 Flash, Claude Opus 4.7, GPT-5.5, Mistral OCR)<n>We release the benchmark, code, and models.
  arXiv  Detail & Related papers  (2026-06-28T05:46:05Z)
• Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions [16.43811675687955]
  Vision-Language Models (VLMs) for optical character recognition can generate plausible but visually unsupported text.<n>We show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions.<n>Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents.
  arXiv  Detail & Related papers  (2026-05-26T22:57:01Z)
• FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing [51.905216364362325]
  We propose FastOCR, a training-free framework with two complementary modules.<n>FastOCR retains 98% of the unpruned model's accuracy while attending to only 5% of the visual tokens per decoding step.
  arXiv  Detail & Related papers  (2026-05-17T13:39:47Z)
• GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts [58.92151016423978]
  We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR across 100+ Unicode scripts.<n>Our benchmark comprises clean and degraded image variants rendered from real multilingual texts.<n>We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts.
  arXiv  Detail & Related papers  (2026-04-14T17:12:41Z)
• Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models [2.1942030377331245]
  Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream?<n>We investigate the OCR routing mechanism across three architecture families using causal interventions.
  arXiv  Detail & Related papers  (2026-02-26T12:06:02Z)
• Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline [58.832237984587664]
  We develop VNLI-Critique, a model for automated sentence-level factuality classification and critique generation.<n>We highlight three key applications: (1) VNLI-Critique demonstrates robust generalization, validated by state-of-the-art performance on the M-HalDetect benchmark; (2) The VNLI-Critique driven AutoRater for DOCCI-Critique provides reliable VLM rankings, showing excellent alignment with human factuality judgments; and (3) An innovative Critic-and-Revise pipeline, achieves substantial improvements in caption factuality.
  arXiv  Detail & Related papers  (2025-06-09T10:57:26Z)
• Vision-centric Token Compression in Large Language Model [51.92055188780033]
  Vision Centric Token Compression (Vist) is a slow-fast compression framework that mirrors human reading.<n>On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%.
  arXiv  Detail & Related papers  (2025-02-02T13:10:06Z)
• CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy [50.78228433498211]
  CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction.<n>It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time.<n>We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition.
  arXiv  Detail & Related papers  (2024-12-03T07:03:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.