Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study

• URL: http://arxiv.org/abs/2606.29213v1
• Date: Sun, 28 Jun 2026 05:46:05 GMT
• Title: Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study
• Abstract summary: OCR systems report strong results on English and Chinese document benchmarks, yet their behaviour on Indic scripts is largely uncharacterised.<n>We benchmark ten systems on Devanagari (Hindi): classical EasyOCR; open VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B, olmOCR-7B); specialised OCR-VLMs (DeepSeek-OCR, Unlimited-OCR); and frontier closed models (Gemini 2.5 Flash, Claude Opus 4.7, GPT-5.5, Mistral OCR)<n>We release the benchmark, code, and models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: OCR systems, ranging from classical engines to specialised OCR vision-language models (OCR-VLMs) and frontier multimodal LLMs, report strong results on English and Chinese document benchmarks, yet their behaviour on Indic scripts is largely uncharacterised. We benchmark ten systems on Devanagari (Hindi): classical EasyOCR; open VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B, olmOCR-7B); specialised OCR-VLMs (DeepSeek-OCR, Unlimited-OCR); and frontier closed models (Gemini 2.5 Flash, Claude Opus 4.7, GPT-5.5, Mistral OCR), across four synthetic degradation conditions and 300 real printed scans. We report four findings. First, on clean rendered text all ten cluster within chrF++ 91 to 98, so synthetic text does not separate them. Second, under degradation the specialised OCR-VLMs are the most fragile: DeepSeek-OCR suffers rare but catastrophic repetition failures (outputs up to 71 the reference length) that wreck its corpus mean even though its median is the best of any system, which is why we report median and catastrophic-rate instead of the mean. Third, on real scans nine of the ten systems collapse (EasyOCR falls from chrF++ 93.6 to 58.3) and the field spreads across a 76-point range, so synthetic renders badly overstate Devanagari quality. Fourth, strong English OCR does not predict Indic OCR: GPT-5.5 drops to chrF++ 58.5 (tying classical EasyOCR) and olmOCR-7B, the model behind olmOCR-Bench, falls to 40.5, while the open Qwen3-VL-8B (75.2, runnable on a single 24 GB GPU) beats GPT-5.5 and approaches Mistral; Gemini and Claude lead at 86.3 and 82.2. An error taxonomy separates surface errors (numerals, punctuation) from structural ones (conjuncts, matras, nukta), and a byte-level (ByT5) post-corrector improves a cheap engine on its own error distribution (chrF++ +1.2 to +1.5) but does not transfer across engines. We release the benchmark, code, and models.

Related papers

• Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewards [44.67582833168009]
  Jina-OCR-v1 is an end-to-end document parsing model built to serve on low-budget GPUs.<n>It combines the compressed-vision encoder and the 3B mixture-of-experts decoder of DeepSeek-OCR.<n>It scores 91.14 on OmniDocBench v1.6 and 83.4 on olmOCR-Bench, and reaches the highest page throughput in our comparison at 2.57 pages per second.
  arXiv  Detail & Related papers  (2026-09-02T21:49:21Z)
• TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents [50.64307290680222]
  TongGuOCR is a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents.<n>TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric.
  arXiv  Detail & Related papers  (2026-08-08T04:50:00Z)
• Cross-Temporal Sinhala OCR: Page-Level Adaptation and Diachronic Analysis [0.0]
  There are no publicly available real-world datasets for page-level Sinhala OCR.<n>We introduce sinhala-ocr-lk-acts-1010, an annotated dataset of 1,010 page-level images and their transcriptions.<n>Three models based on deep learning-based visual language processing are fine-tuned using QLoRA.
  arXiv  Detail & Related papers  (2026-06-28T13:01:54Z)
• How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations [17.186250064889887]
  Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks, but their robustness under controlled visual degradation remains insufficiently understood.<n>We introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations.<n>We conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability.
  arXiv  Detail & Related papers  (2026-06-24T17:15:42Z)
• Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models [10.628090041273929]
  Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding.<n>We show they do not always act as faithful transcribers: when text is imperfect, they often rewrite it into a more plausible form.
  arXiv  Detail & Related papers  (2026-05-29T20:33:35Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts [58.92151016423978]
  We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR across 100+ Unicode scripts.<n>Our benchmark comprises clean and degraded image variants rendered from real multilingual texts.<n>We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts.
  arXiv  Detail & Related papers  (2026-04-14T17:12:41Z)
• Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models [2.1942030377331245]
  Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream?<n>We investigate the OCR routing mechanism across three architecture families using causal interventions.
  arXiv  Detail & Related papers  (2026-02-26T12:06:02Z)
• GutenOCR: A Grounded Vision-Language Front-End for Documents [0.42776193697639947]
  We introduce a family of grounded OCR front-ends obtained by fine-tuning Qwen2.5-VL-3B and Qwen2.5-VL-7B.<n>The resulting single-checkpoint vision-language models expose reading, detection, and grounding through a unified, prompt-based interface.<n>We show that GutenOCR-7B more than doubles the composite grounded OCR score of its Qwen2.5-VL-7B backbone on 10.5K held-out business and scientific pages.
  arXiv  Detail & Related papers  (2026-01-20T21:26:15Z)
• Finetuning Vision-Language Models as OCR Systems for Low-Resource Languages: A Case Study of Manchu [0.0]
  Manchu, a critically endangered language, lacks effective OCR systems that can handle real-world historical documents.<n>This study develops high-performing OCR systems by fine-tuning three open-source vision-language models.<n>LLaMA-3.2-11B achieved exceptional performance with 98.3% word accuracy and 0.0024 character error rate on synthetic data.
  arXiv  Detail & Related papers  (2025-07-09T11:38:20Z)
• MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
  MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
  arXiv  Detail & Related papers  (2025-06-05T16:34:57Z)
• CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy [50.78228433498211]
  CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction.<n>It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time.<n>We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition.
  arXiv  Detail & Related papers  (2024-12-03T07:03:25Z)
• LOCR: Location-Guided Transformer for Optical Character Recognition [55.195165959662795]
  We propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.
  arXiv  Detail & Related papers  (2024-03-04T15:34:12Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.