Learning Dense Representations of Phrases at Scale

• URL: http://arxiv.org/abs/2012.12624v2
• Date: Sat, 2 Jan 2021 00:42:50 GMT
• Title: Learning Dense Representations of Phrases at Scale
• Authors: Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, Danqi Chen
• Abstract summary: We show for the first time that we can learn dense phrase representations alone that achieve much stronger performance in open-domain QA. Our model DensePhrases improves previous phrase retrieval models by 15%-25% absolute accuracy. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs.
• Score: 22.792942611601347
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on their sparse representations while still underperforming retriever-reader approaches. In this work, we show for the first time that we can learn dense phrase representations alone that achieve much stronger performance in open-domain QA. Our approach includes (1) learning query-agnostic phrase representations via question generation and distillation; (2) novel negative-sampling methods for global normalization; (3) query-side fine-tuning for transfer learning. On five popular QA datasets, our model DensePhrases improves previous phrase retrieval models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.

Related papers

• Efficient Prompt Caching via Embedding Similarity [26.456212783693545]
  We focus on the prediction accuracy of prompt caching for single-round question-answering tasks via embedding similarity. We propose a distillation-based method to fine-tune the existing embeddings for better better prediction. We also conduct simulations demonstrating that our trained models achieve better caching efficiency than the previous embedding model.
  arXiv  Detail & Related papers  (2024-02-02T06:34:11Z)
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context [55.564789967211844]
  An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant. Recent work has shown that retrieval augmentation can sometimes have a negative effect on performance.
  arXiv  Detail & Related papers  (2023-10-02T18:52:35Z)
• Bridging the Training-Inference Gap for Dense Phrase Retrieval [104.4836127502683]
  Building dense retrievers requires a series of standard procedures, including training and validating neural models. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced. We propose an efficient way of validating dense retrievers using a small subset of the entire corpus.
  arXiv  Detail & Related papers  (2022-10-25T00:53:06Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• Improving Passage Retrieval with Zero-Shot Question Generation [109.11542468380331]
  We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage.
  arXiv  Detail & Related papers  (2022-04-15T14:51:41Z)
• RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing [11.157958012672202]
  We are applying retrieval-based modeling ideas to the problem of multi-domain task-oriented semantic parsing. Our approach, RetroNLU, extends a sequence-to-sequence model architecture with a retrieval component. We analyze the nearest neighbor retrieval component's quality, model sensitivity and break down the performance for semantic parses of different utterance complexity.
  arXiv  Detail & Related papers  (2021-09-21T19:30:30Z)
• Phrase Retrieval Learns Passage Retrieval, Too [77.57208968326422]
  We study whether phrase retrieval can serve as the basis for coarse-level retrieval including passages and documents. We show that a dense phrase-retrieval system, without any retraining, already achieves better passage retrieval accuracy. We also show that phrase filtering and vector quantization can reduce the size of our index by 4-10x.
  arXiv  Detail & Related papers  (2021-09-16T17:42:45Z)
• EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System [0.0]
  In this paper, we explore the possibility to transfer the natural language understanding of language models into dense vectors representing questions and answer candidates. Our model achieves state-of-the-art results in Phrase-Indexed Question Answering (PIQA) beating the previous state-of-art by 1.3 points in exact-match and 1.4 points in f1-score.
  arXiv  Detail & Related papers  (2021-01-06T17:46:05Z)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks [133.93803565077337]
  retrieval-augmented generation models combine pre-trained parametric and non-parametric memory for language generation. We show that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
  arXiv  Detail & Related papers  (2020-05-22T21:34:34Z)
• REALM: Retrieval-Augmented Language Model Pre-Training [37.3178586179607]
  We augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA)
  arXiv  Detail & Related papers  (2020-02-10T18:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.