KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR

• URL: http://arxiv.org/abs/2603.13238v1
• Date: Tue, 17 Feb 2026 14:24:49 GMT
• Title: KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR
• Abstract summary: Kazakh is a Turkic language using the Arabic, Cyrillic, and Latin scripts, making it unique in terms of optical character recognition (OCR)<n>Work on OCR for low-resource Kazakh scripts is very scarce, and no OCR benchmarks or images exist for the Arabic and Latin scripts.<n>We construct a synthetic OCR dataset of 7,219 images for all three scripts with font, color, and noise variations to imitate real OCR tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Kazakh is a Turkic language using the Arabic, Cyrillic, and Latin scripts, making it unique in terms of optical character recognition (OCR). Work on OCR for low-resource Kazakh scripts is very scarce, and no OCR benchmarks or images exist for the Arabic and Latin scripts. We construct a synthetic OCR dataset of 7,219 images for all three scripts with font, color, and noise variations to imitate real OCR tasks. We evaluated three multimodal large language models (MLLMs) on a subset of the benchmark for OCR and language identification: Gemma-3-12B-it, Qwen2.5-VL-7B-Instruct, and Llama-3.2-11B-Vision-Instruct. All models are unsuccessful with Latin and Arabic script OCR, and fail to recognize the Arabic script as Kazakh text, misclassifying it as Arabic, Farsi, and Kurdish. We further compare MLLMs with a classical OCR baseline and find that while traditional OCR has lower character error rates, MLLMs fail to match this performance. These findings show significant gaps in current MLLM capabilities to process low-resource Abjad-based scripts and demonstrate the need for inclusive models and benchmarks supporting low-resource scripts and languages.

Related papers

• When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study [9.108791469266345]
  We evaluate traditional OCR, general-purpose VLMs, Arabic-specialized VLMs, and OCR-conditioned VLM correction across eight Arabic text datasets.<n> OCR-prior recoverability helps when the OCR output remains visually and textually recoverable, providing anchors that the VLM can refine against the image.<n>Additional diagnostics show that Arabic VLM-OCR datasets is sensitive to diacritics, preprocessing, generation budget, and repetition loops.
  arXiv  Detail & Related papers  (2026-08-23T11:12:06Z)
• TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents [50.64307290680222]
  TongGuOCR is a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents.<n>TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric.
  arXiv  Detail & Related papers  (2026-08-08T04:50:00Z)
• GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts [58.92151016423978]
  We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR across 100+ Unicode scripts.<n>Our benchmark comprises clean and degraded image variants rendered from real multilingual texts.<n>We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts.
  arXiv  Detail & Related papers  (2026-04-14T17:12:41Z)
• LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families [53.111005425237245]
  Large language models (LLMs) have driven substantial advances in speech language models (SpeechLMs)<n>LoASR-Bench comprises 25 languages from 9 language families, featuring both Latin and non-Latin scripts.<n> Experimental results highlight the limitations of the latest SpeechLMs in handling real-world low-resource languages.
  arXiv  Detail & Related papers  (2026-03-20T15:26:34Z)
• Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR [1.7590081165362783]
  We introduce Baseer, a vision-language model fine- tuned specifically for Arabic document OCR.<n>Leveraging a large-scale dataset combining synthetic and real-world documents, Baseer is trained using a decoder-only fine-tuning strategy.<n>Our experiments show that Baseer significantly outperforms existing open-source and commercial solutions, achieving a WER of 0.25.
  arXiv  Detail & Related papers  (2025-09-17T15:07:29Z)
• QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation [0.8944616102795021]
  We present Qari-OCR, a vision-language models progressively optimized for Arabic.<n>Qari-OCR establishes a new open-source state-of-the-art with a Word Error Rate (WER) of 0.160, Character Error Rate (CER) of 0.061, and BLEU score of 0.737 on diacritically-rich texts.
  arXiv  Detail & Related papers  (2025-06-02T22:21:06Z)
• KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding [24.9462694200992]
  KITAB-Bench is a comprehensive Arabic OCR benchmark that fills the gaps in current evaluation systems.<n>Modern vision-language models (such as GPT-4o, Gemini, and Qwen) outperform traditional OCR approaches by an average of 60% in Character Error Rate (CER)<n>This work establishes a rigorous evaluation framework that can drive improvements in Arabic document analysis methods.
  arXiv  Detail & Related papers  (2025-02-20T18:41:23Z)
• Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts [0.0]
  This study investigates the potential of Large Language Models (LLMs), particularly GPT-4o, for Optical Character Recognition (OCR) in low-resource scripts such as Urdu, Albanian, and Tajik.<n>Using a meticulously curated dataset of 2,520 images incorporating controlled variations in text length, font size, background color, and blur, the research simulates diverse real-world challenges.
  arXiv  Detail & Related papers  (2024-12-20T18:05:22Z)
• CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy [50.78228433498211]
  CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction.<n>It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time.<n>We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition.
  arXiv  Detail & Related papers  (2024-12-03T07:03:25Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language.<n>Currently, instruction-tuned large language models (LLMs) excel at various English tasks.<n>Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts [50.40191599304911]
  We investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. We propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. Our findings show that the effectiveness of transliteration varies by task type and model size.
  arXiv  Detail & Related papers  (2024-07-02T14:51:20Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.