Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval

• URL: http://arxiv.org/abs/2505.05666v1
• Date: Thu, 08 May 2025 21:54:02 GMT
• Title: Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval
• Authors: Alexander Most, Joseph Winjum, Ayan Biswas, Shawn Jones, Nishath Rajiv Ranasinghe, Dan O'Malley, Manish Bhattarai,
• Abstract summary: Retrieval-Augmented Generation (RAG) is a technique for grounding responses in external documents.<n>Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text.<n>Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR.
• Score: 38.569818461453394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-Augmented Generation (RAG) has become a popular technique for enhancing the reliability and utility of Large Language Models (LLMs) by grounding responses in external documents. Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text. However, even state-of-the-art OCRs can introduce errors, especially in degraded or complex documents. Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR. This study presents a systematic comparison between a vision-based RAG system (ColPali) and more traditional OCR-based pipelines utilizing Llama 3.2 (90B) and Nougat OCR across varying document qualities. Beyond conventional retrieval accuracy metrics, we introduce a semantic answer evaluation benchmark to assess end-to-end question-answering performance. Our findings indicate that while vision-based RAG performs well on documents it has been fine-tuned on, OCR-based RAG is better able to generalize to unseen documents of varying quality. We highlight the key trade-offs between computational efficiency and semantic accuracy, offering practical guidance for RAG practitioners in selecting between OCR-dependent and vision-based document retrieval systems in production environments.

Related papers

• Controlled Retrieval-augmented Context Evaluation for Long-form RAG [58.14561461943611]
  Retrieval-augmented generation (RAG) enhances large language models by incorporating context retrieved from external knowledge sources.<n>We argue that providing a comprehensive retrieval-augmented context is important for long-form RAG tasks like report generation.<n>We introduce CRUX, a framework designed to directly assess retrieval-augmented contexts.
  arXiv  Detail & Related papers  (2025-06-24T23:17:48Z)
• Words as Geometric Features: Estimating Homography using Optical Character Recognition as Compressed Image Representation [6.385732495789276]
  Document alignment plays a crucial role in numerous real-world applications, such as automated form processing, anomaly detection, and workflow automation.<n>Traditional methods for document alignment rely on image-based features like keypoints, edges, and textures to estimate geometric transformations, such as homographies.<n>This paper introduces a novel approach that leverages Optical Character Recognition (OCR) outputs as features for homography estimation.
  arXiv  Detail & Related papers  (2025-05-25T01:20:32Z)
• VISTA-OCR: Towards generative and interactive end to end OCR models [3.7548609506798494]
  VISTA-OCR is a lightweight architecture that unifies text detection and recognition within a single generative model.<n>Built on an encoder-decoder architecture, VISTA-OCR is progressively trained, starting with the visual feature extraction phase.<n>To enhance the model's capabilities, we built a new dataset composed of real-world examples enriched with bounding box annotations and synthetic samples.
  arXiv  Detail & Related papers  (2025-04-04T17:39:53Z)
• OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation [27.897982337072335]
  Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining.<n>As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR)<n>In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems.
  arXiv  Detail & Related papers  (2024-12-03T17:23:47Z)
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents [66.42579289213941]
  Retrieval-augmented generation (RAG) is an effective technique that enables large language models to utilize external knowledge sources for generation.<n>We introduce VisRAG, which tackles this issue by establishing a vision-language model (VLM)-based RAG pipeline.<n>In this pipeline, instead of first parsing the document to obtain text, the document is directly embedded using a VLM as an image and then retrieved to enhance the generation of a VLM.
  arXiv  Detail & Related papers  (2024-10-14T15:04:18Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• User-Centric Evaluation of OCR Systems for Kwak'wala [92.73847703011353]
  We show that utilizing OCR reduces the time spent in the manual transcription of culturally valuable documents by over 50%. Our results demonstrate the potential benefits that OCR tools can have on downstream language documentation and revitalization efforts.
  arXiv  Detail & Related papers  (2023-02-26T21:41:15Z)
• Donut: Document Understanding Transformer without OCR [17.397447819420695]
  We propose a novel VDU model that is end-to-end trainable without underpinning OCR framework. Our approach achieves state-of-the-art performance on various document understanding tasks in public benchmark datasets and private industrial service datasets.
  arXiv  Detail & Related papers  (2021-11-30T18:55:19Z)
• Lights, Camera, Action! A Framework to Improve NLP Accuracy over OCR documents [2.6201102730518606]
  We demonstrate an effective framework for mitigating OCR errors for any downstream NLP task. We first address the data scarcity problem for model training by constructing a document synthesis pipeline. For the benefit of the community, we have made the document synthesis pipeline available as an open-source project.
  arXiv  Detail & Related papers  (2021-08-06T00:32:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.