Exploring OCR-augmented Generation for Bilingual VQA

• URL: http://arxiv.org/abs/2510.02543v1
• Date: Thu, 02 Oct 2025 20:19:31 GMT
• Title: Exploring OCR-augmented Generation for Bilingual VQA
• Authors: JoonHo Lee, Sunho Park,
• Abstract summary: We investigate OCR-augmented generation with Vision Language Models (VLMs)<n>To support research in this domain, we train and release KLOCR.<n>To complement existing VQA benchmarks, we curate KOCRBench for Korean VQA.
• Score: 5.541485330114692
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We investigate OCR-augmented generation with Vision Language Models (VLMs), exploring tasks in Korean and English toward multilingualism. To support research in this domain, we train and release KLOCR, a strong bilingual OCR baseline trained on 100M instances to augment VLMs with OCR ability. To complement existing VQA benchmarks, we curate KOCRBench for Korean VQA, and analyze different prompting methods. Extensive experiments show that OCR-extracted text significantly boosts performance across open source and commercial models. Our work offers new insights into OCR-augmented generation for bilingual VQA. Model, code, and data are available at https://github.com/JHLee0513/KLOCR.

Related papers

• Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency [31.095908827004695]
  Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks.<n>They struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges.<n>We introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage"
  arXiv  Detail & Related papers  (2025-07-11T05:02:06Z)
• Evaluating Robustness of LLMs in Question Answering on Multilingual Noisy OCR Data [26.34532500908141]
  We conduct a comprehensive analysis of how OCR-induced noise affects the performance of Multilingual QA Systems.<n>To support this analysis, we introduce a multilingual QA dataset MultiOCR-QA, comprising 50K question-answer pairs across three languages.<n>The dataset is curated from OCR-ed historical documents, which include different levels and types of OCR noise.<n>We then evaluate how different state-of-the-art Large Language Models (LLMs) perform under different error conditions, focusing on three major OCR error types.
  arXiv  Detail & Related papers  (2025-02-24T02:16:37Z)
• Ocean-OCR: Towards General OCR Application via a Vision-Language Model [6.70908296002235]
  We present textbfOcean-OCR, a 3B MLLM with state-of-the-art performance on various OCR scenarios and comparable understanding ability on general tasks.<n>We demonstrate the superiority of Ocean-OCR through comprehensive experiments on open-source OCR benchmarks and across various OCR scenarios.
  arXiv  Detail & Related papers  (2025-01-26T15:20:39Z)
• CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy [50.78228433498211]
  CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction.<n>It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time.<n>We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition.
  arXiv  Detail & Related papers  (2024-12-03T07:03:25Z)
• Spanish TrOCR: Leveraging Transfer Learning for Language Adaptation [0.0]
  This study explores the transfer learning capabilities of the TrOCR architecture to Spanish. We integrate an English TrOCR encoder with a language specific decoder and train the model on this specific language. Fine-tuning the English TrOCR on Spanish yields superior recognition than the language specific decoder for a fixed dataset size.
  arXiv  Detail & Related papers  (2024-07-09T15:31:41Z)
• Text-Conditioned Resampler For Long Form Video Understanding [94.81955667020867]
  We present a text-conditioned video resampler (TCR) module that uses a pre-trained visual encoder and large language model (LLM) TCR can process more than 100 frames at a time with plain attention and without optimised implementations.
  arXiv  Detail & Related papers  (2023-12-19T06:42:47Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)
• OCR Improves Machine Translation for Low-Resource Languages [10.010595434359647]
  We introduce and make publicly available a novel benchmark, textscOCR4MT, consisting of real and synthetic data, enriched with noise. We evaluate state-of-the-art OCR systems on our benchmark and analyse most common errors. We then perform an ablation study to investigate how OCR errors impact Machine Translation performance.
  arXiv  Detail & Related papers  (2022-02-27T02:36:45Z)
• Structured Multimodal Attentions for TextVQA [57.71060302874151]
  We propose an end-to-end structured multimodal attention (SMA) neural network to mainly solve the first two issues above. SMA first uses a structural graph representation to encode the object-object, object-text and text-text relationships appearing in the image, and then designs a multimodal graph attention network to reason over it. Our proposed model outperforms the SoTA models on TextVQA dataset and two tasks of ST-VQA dataset among all models except pre-training based TAP.
  arXiv  Detail & Related papers  (2020-06-01T07:07:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.