DAPR: A Benchmark on Document-Aware Passage Retrieval

• URL: http://arxiv.org/abs/2305.13915v4
• Date: Sun, 9 Jun 2024 16:00:37 GMT
• Title: DAPR: A Benchmark on Document-Aware Passage Retrieval
• Authors: Kexin Wang, Nils Reimers, Iryna Gurevych,
• Abstract summary: We propose and name this task emphDocument-Aware Passage Retrieval (DAPR) While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
• Score: 57.45793782107218
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The work of neural retrieval so far focuses on ranking short texts and is challenged with long documents. There are many cases where the users want to find a relevant passage within a long document from a huge corpus, e.g. Wikipedia articles, research papers, etc. We propose and name this task \emph{Document-Aware Passage Retrieval} (DAPR). While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5\%) are due to missing document context. This drives us to build a benchmark for this task including multiple datasets from heterogeneous domains. In the experiments, we extend the SoTA passage retrievers with document context via (1) hybrid retrieval with BM25 and (2) contextualized passage representations, which inform the passage representation with document context. We find despite that hybrid retrieval performs the strongest on the mixture of the easy and the hard queries, it completely fails on the hard queries that require document-context understanding. On the other hand, contextualized passage representations (e.g. prepending document titles) achieve good improvement on these hard queries, but overall they also perform rather poorly. Our created benchmark enables future research on developing and comparing retrieval systems for the new task. The code and the data are available at https://github.com/UKPLab/arxiv2023-dapr.

Related papers

• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  Many complex real-world queries require in-depth reasoning to identify relevant documents. We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• PDFTriage: Question Answering over Long, Structured Documents [60.96667912964659]
  Representing structured documents as plain text is incongruous with the user's mental model of these documents with rich structure. We propose PDFTriage that enables models to retrieve the context based on either structure or content. Our benchmark dataset consists of 900+ human-generated questions over 80 structured documents.
  arXiv  Detail & Related papers  (2023-09-16T04:29:05Z)
• Fine-Grained Distillation for Long Document Retrieval [86.39802110609062]
  Long document retrieval aims to fetch query-relevant documents from a large-scale collection. Knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. We propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers.
  arXiv  Detail & Related papers  (2022-12-20T17:00:36Z)
• CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion [68.19934563919192]
  We propose a curriculum sampling strategy that utilizes pseudo queries during training and progressively enhances the relevance between the generated query and the real query. Experimental results on both in-domain and out-of-domain datasets demonstrate that our approach outperforms previous dense retrieval models.
  arXiv  Detail & Related papers  (2022-12-18T15:57:46Z)
• Generate rather than Retrieve: Large Language Models are Strong Context Generators [74.87021992611672]
  We present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer.
  arXiv  Detail & Related papers  (2022-09-21T01:30:59Z)
• Few-Shot Document-Level Event Argument Extraction [2.680014762694412]
  Event argument extraction (EAE) has been well studied at the sentence level but under-explored at the document level. We present FewDocAE, a Few-Shot Document-Level Event Argument Extraction benchmark.
  arXiv  Detail & Related papers  (2022-09-06T03:57:23Z)
• Query-Based Keyphrase Extraction from Long Documents [4.823229052465654]
  This paper overcomes issue for keyphrase extraction by chunking the long documents. System employs a pre-trained BERT model and adapts it to estimate the probability that a given text span forms a keyphrase.
  arXiv  Detail & Related papers  (2022-05-11T10:29:30Z)
• CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Example [43.01717754418893]
  We introduce the task of faceted Query by Example. Users can also specify a finer grained aspect in addition to the input query document. We envision models which are able to retrieve scientific papers analogous to a query scientific paper.
  arXiv  Detail & Related papers  (2021-03-24T01:02:12Z)
• Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering [9.480648914353035]
  We present FiRA: a novel dataset of Fine-Grained Relevances. We extend the ranked retrieval annotations of the Deep Learning track of TREC 2019 with passage and word level graded relevance annotations for all relevant documents. As an example, we evaluate the recently introduced TKL document ranking model. We find that although TKL exhibits state-of-the-art retrieval results for long documents, it misses many relevant passages.
  arXiv  Detail & Related papers  (2020-08-12T14:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.