LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval

• URL: http://arxiv.org/abs/2208.13661v1
• Date: Mon, 29 Aug 2022 15:09:28 GMT
• Title: LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval
• Authors: Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, Daxin Jiang
• Abstract summary: We propose to make a dense retriever align a well-performing lexicon-aware representation model. We evaluate our model on three public benchmarks, which shows that with a comparable lexicon-aware retriever as the teacher, our proposed dense model can bring consistent and significant improvements.
• Score: 68.85686621130111
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval models based on dense representations in semantic space have become an indispensable branch for first-stage retrieval. These retrievers benefit from surging advances in representation learning towards compressive global sequence-level embeddings. However, they are prone to overlook local salient phrases and entity mentions in texts, which usually play pivot roles in first-stage retrieval. To mitigate this weakness, we propose to make a dense retriever align a well-performing lexicon-aware representation model. The alignment is achieved by weakened knowledge distillations to enlighten the retriever via two aspects -- 1) a lexicon-augmented contrastive objective to challenge the dense encoder and 2) a pair-wise rank-consistent regularization to make dense model's behavior incline to the other. We evaluate our model on three public benchmarks, which shows that with a comparable lexicon-aware retriever as the teacher, our proposed dense one can bring consistent and significant improvements, and even outdo its teacher. In addition, we found our improvement on the dense retriever is complementary to the standard ranker distillation, which can further lift state-of-the-art performance.

Related papers

• Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers [6.773411876899064]
  inference-free sparse models lag far behind in terms of search relevance when compared to both sparse and dense siamese models. We propose two different approaches for performance improvement. First, we introduce the IDF-aware FLOPS loss, which introduces Inverted Document Frequency (IDF) to the sparsification of representations. We find that it mitigates the negative impact of the FLOPS regularization on search relevance, allowing the model to achieve a better balance between accuracy and efficiency.
  arXiv  Detail & Related papers  (2024-11-07T03:46:43Z)
• Bridging the Training-Inference Gap for Dense Phrase Retrieval [104.4836127502683]
  Building dense retrievers requires a series of standard procedures, including training and validating neural models. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced. We propose an efficient way of validating dense retrievers using a small subset of the entire corpus.
  arXiv  Detail & Related papers  (2022-10-25T00:53:06Z)
• Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning [48.18060169551869]
  We propose a metric distillation rule to distill knowledge from the metric to the ranker. We further transfer the critical knowledge summarized by the distilled ranker to the retriever. Experimental results on the CommonGen benchmark verify the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2022-10-21T03:34:24Z)
• Towards Robust Ranker for Text Retrieval [83.15191578888188]
  A ranker plays an indispensable role in the de facto'retrieval & rerank' pipeline. A ranker plays an indispensable role in the de facto'retrieval & rerank' pipeline.
  arXiv  Detail & Related papers  (2022-06-16T10:27:46Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval [55.097573036580066]
  Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.
  arXiv  Detail & Related papers  (2022-03-11T18:53:12Z)
• Adversarial Retriever-Ranker for dense text retrieval [51.87158529880056]
  We present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker. AR2 consistently and significantly outperforms existing dense retriever methods. This includes the improvements on Natural Questions R@5 to 77.9%(+2.1%), TriviaQA R@5 to 78.2%(+1.4), and MS-MARCO MRR@10 to 39.5%(+1.3%)
  arXiv  Detail & Related papers  (2021-10-07T16:41:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.