Saving Dense Retriever from Shortcut Dependency in Conversational Search

• URL: http://arxiv.org/abs/2202.07280v1
• Date: Tue, 15 Feb 2022 09:53:35 GMT
• Title: Saving Dense Retriever from Shortcut Dependency in Conversational Search
• Authors: Sungdong Kim, Gangwoo Kim
• Abstract summary: A retrieval shortcut in conversational search (CS) causes models to retrieve passages solely relying on partial history while disregarding the latest question. We show naively trained dense retrievers heavily exploit the shortcut and hence perform poorly when asked to answer history-independent questions. To prevent models from solely relying on the shortcut, we explore iterative hard negatives mined by pre-trained dense retrievers.
• Score: 7.584170081762014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In conversational search (CS), it needs holistic understanding over conversational inputs to retrieve relevant passages. In this paper, we demonstrate the existence of a retrieval shortcut in CS, which causes models to retrieve passages solely relying on partial history while disregarding the latest question. With in-depth analysis, we first show naively trained dense retrievers heavily exploit the shortcut and hence perform poorly when asked to answer history-independent questions. To prevent models from solely relying on the shortcut, we explore iterative hard negatives mined by pre-trained dense retrievers. Experimental results show that training with the iterative hard negatives effectively mitigates the dependency on the shortcut and makes substantial improvement on recent CS benchmarks. Our retrievers achieve new state-of-the-art results, outperforming the previous best models by 9.7 in Recall@10 on QReCC and 12.4 in Recall@5 on TopiOCQA. Furthermore, in our end-to-end QA experiments, FiD readers combined with our retrievers surpass the previous state-of-the-art models by 3.7 and 1.0 EM scores on QReCC and TopiOCQA, respectively.

Related papers

• ReasonIR: Training Retrievers for Reasoning Tasks [139.54343970560103]
  ReasonIR-8B is the first retriever specifically trained for general reasoning tasks. It achieves a new state-of-the-art of 29.9 nDCG@10 without reranker and 36.9 nDCG@10 with reranker on BRIGHT, a widely-used information retrieval benchmark.
  arXiv  Detail & Related papers  (2025-04-29T09:49:28Z)
• Efficient Long-range Language Modeling with Self-supervised Causal Retrieval [39.24972628990943]
  Grouped Cross-Attention is a novel module enabling joint pre-training of the retriever and causal LM. By integrating top-$k$ retrieval, our model can be pre-trained efficiently from scratch with context lengths up to 64K tokens.
  arXiv  Detail & Related papers  (2024-10-02T15:18:34Z)
• Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversational Search [25.16282868262589]
  RetPO is designed to optimize a language model (LM) for reformulating search queries in line with the preferences of the target retrieval systems. We construct a large-scale dataset called Retrievers' Feedback on over 410K query rewrites across 12K conversations. The resulting model achieves state-of-the-art performance on two recent conversational search benchmarks.
  arXiv  Detail & Related papers  (2024-02-19T04:41:31Z)
• Phrase Retrieval for Open-Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive Learning [54.55643652781891]
  Open-Domain Conversational Question Answering (ODConvQA) aims at answering questions through a multi-turn conversation. We propose a method to directly predict answers with a phrase retrieval scheme for a sequence of words.
  arXiv  Detail & Related papers  (2023-06-07T09:46:38Z)
• ReFIT: Relevance Feedback from a Reranker during Inference [109.33278799999582]
  Retrieve-and-rerank is a prevalent framework in neural information retrieval. We propose to leverage the reranker to improve recall by making it provide relevance feedback to the retriever at inference time.
  arXiv  Detail & Related papers  (2023-05-19T15:30:33Z)
• Improving Passage Retrieval with Zero-Shot Question Generation [109.11542468380331]
  We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage.
  arXiv  Detail & Related papers  (2022-04-15T14:51:41Z)
• Adversarial Retriever-Ranker for dense text retrieval [51.87158529880056]
  We present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker. AR2 consistently and significantly outperforms existing dense retriever methods. This includes the improvements on Natural Questions R@5 to 77.9%(+2.1%), TriviaQA R@5 to 78.2%(+1.4), and MS-MARCO MRR@10 to 39.5%(+1.3%)
  arXiv  Detail & Related papers  (2021-10-07T16:41:15Z)
• End-to-End Training of Neural Retrievers for Open-Domain Question Answering [32.747113232867825]
  It remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers. We propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans. We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
  arXiv  Detail & Related papers  (2021-01-02T09:05:34Z)
• RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering [49.024513062811685]
  We develop a simple and effective re-ranking approach (RECONSIDER) for span-extraction tasks. RECONSIDER is trained on positive and negative examples extracted from high confidence predictions of MRC models. It uses in-passage span annotations to perform span-focused re-ranking over a smaller candidate set.
  arXiv  Detail & Related papers  (2020-10-21T04:28:42Z)
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA [95.9105154311491]
  We introduce two approaches to improve unsupervised Question Answering (QA) First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA) Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA.
  arXiv  Detail & Related papers  (2020-05-06T15:56:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.