Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction

• URL: http://arxiv.org/abs/2505.07730v1
• Date: Mon, 12 May 2025 16:37:47 GMT
• Title: Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction
• Authors: Jingfen Qiao, Jia-Huei Ju, Xinyu Ma, Evangelos Kanoulas, Andrew Yates,
• Abstract summary: Visual Document Retrieval (VDR) is an emerging research area that focuses on encoding and retrieving document images directly.<n>A recent advance in VDR was introduced by ColPali, which significantly improved retrieval effectiveness through a late interaction mechanism.<n>Our research investigates the specific contributions of late interaction by looking into query-patch matching in the context of visual document retrieval.
• Score: 22.41501622100226
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual Document Retrieval (VDR) is an emerging research area that focuses on encoding and retrieving document images directly, bypassing the dependence on Optical Character Recognition (OCR) for document search. A recent advance in VDR was introduced by ColPali, which significantly improved retrieval effectiveness through a late interaction mechanism. ColPali's approach demonstrated substantial performance gains over existing baselines that do not use late interaction on an established benchmark. In this study, we investigate the reproducibility and replicability of VDR methods with and without late interaction mechanisms by systematically evaluating their performance across multiple pre-trained vision-language models. Our findings confirm that late interaction yields considerable improvements in retrieval effectiveness; however, it also introduces computational inefficiencies during inference. Additionally, we examine the adaptability of VDR models to textual inputs and assess their robustness across text-intensive datasets within the proposed benchmark, particularly when scaling the indexing mechanism. Furthermore, our research investigates the specific contributions of late interaction by looking into query-patch matching in the context of visual document retrieval. We find that although query tokens cannot explicitly match image patches as in the text retrieval scenario, they tend to match the patch contains visually similar tokens or their surrounding patches.

Related papers

• From Ambiguity to Accuracy: The Transformative Effect of Coreference Resolution on Retrieval-Augmented Generation systems [6.762635083456022]
  We investigate how entity coreference affects both document retrieval and generative performance in RAG-based systems.<n>We demonstrate that coreference resolution enhances retrieval effectiveness and improves question-answering (QA) performance.<n>This study aims to provide a deeper understanding of the challenges posed by coreferential complexity in RAG, providing guidance for improving retrieval and generation in knowledge-intensive AI applications.
  arXiv  Detail & Related papers  (2025-07-10T15:26:59Z)
• Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval [38.569818461453394]
  Retrieval-Augmented Generation (RAG) is a technique for grounding responses in external documents.<n>Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text.<n>Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR.
  arXiv  Detail & Related papers  (2025-05-08T21:54:02Z)
• Generalized Visual Relation Detection with Diffusion Models [94.62313788626128]
  Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image.<n>We propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner.<n>Our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets.
  arXiv  Detail & Related papers  (2025-04-16T14:03:24Z)
• Exploring Information Retrieval Landscapes: An Investigation of a Novel Evaluation Techniques and Comparative Document Splitting Methods [0.0]
  In this study, the structured nature of textbooks, the conciseness of articles, and the narrative complexity of novels are shown to require distinct retrieval strategies. A novel evaluation technique is introduced, utilizing an open-source model to generate a comprehensive dataset of question-and-answer pairs. The evaluation employs weighted scoring metrics, including SequenceMatcher, BLEU, METEOR, and BERT Score, to assess the system's accuracy and relevance.
  arXiv  Detail & Related papers  (2024-09-13T02:08:47Z)
• Enhancing Retrieval-Augmented LMs with a Two-stage Consistency Learning Compressor [4.35807211471107]
  This work proposes a novel two-stage consistency learning approach for retrieved information compression in retrieval-augmented language models. The proposed method is empirically validated across multiple datasets, demonstrating notable enhancements in precision and efficiency for question-answering tasks.
  arXiv  Detail & Related papers  (2024-06-04T12:43:23Z)
• Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness [56.42192735214931]
  retrievers are expected to not only rely on the semantic relevance between the documents and the queries but also recognize the nuanced intents or perspectives behind a user query. In this work, we study whether retrievers can recognize and respond to different perspectives of the queries. We show that current retrievers have limited awareness of subtly different perspectives in queries and can also be biased toward certain perspectives.
  arXiv  Detail & Related papers  (2024-05-04T17:10:00Z)
• Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models [58.17315970207874]
  We propose a zero-shot method for adapting generalisable visual-textual priors from arbitrary VLM to facilitate moment-text alignment. Experiments conducted on three VMR benchmark datasets demonstrate the notable performance advantages of our zero-shot algorithm.
  arXiv  Detail & Related papers  (2023-09-01T13:06:50Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Where Does the Performance Improvement Come From? - A Reproducibility Concern about Image-Text Retrieval [85.03655458677295]
  Image-text retrieval has gradually become a major research direction in the field of information retrieval. We first examine the related concerns and why the focus is on image-text retrieval tasks. We analyze various aspects of the reproduction of pretrained and nonpretrained retrieval models.
  arXiv  Detail & Related papers  (2022-03-08T05:01:43Z)
• Generation-Augmented Retrieval for Open-domain Question Answering [134.27768711201202]
  Generation-Augmented Retrieval (GAR) for answering open-domain questions. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader.
  arXiv  Detail & Related papers  (2020-09-17T23:08:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.