Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering

• URL: http://arxiv.org/abs/2110.01599v1
• Date: Mon, 4 Oct 2021 17:51:07 GMT
• Title: Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering
• Authors: Minghan Li, Jimmy Lin
• Abstract summary: We study how an in-distribution (IND) question/passage encoder would generalize if paired with an OOD passage/question encoder from another domain. We find that the passage encoder has more influence on the lower bound of generalization while the question encoder seems to affect the upper bound in general.
• Score: 44.853870854321066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One key feature of dense passage retrievers (DPR) is the use of separate question and passage encoder in a bi-encoder design. Previous work on generalization of DPR mainly focus on testing both encoders in tandem on out-of-distribution (OOD) question-answering (QA) tasks, which is also known as domain adaptation. However, it is still unknown how DPR's individual question/passage encoder affects generalization. Specifically, in this paper, we want to know how an in-distribution (IND) question/passage encoder would generalize if paired with an OOD passage/question encoder from another domain. We refer to this challenge as \textit{encoder adaptation}. To answer this question, we inspect different combinations of DPR's question and passage encoder learned from five benchmark QA datasets on both in-domain and out-of-domain questions. We find that the passage encoder has more influence on the lower bound of generalization while the question encoder seems to affect the upper bound in general. For example, applying an OOD passage encoder usually hurts the retrieval accuracy while an OOD question encoder sometimes even improves the accuracy.

Related papers

• Hold Me Tight: Stable Encoder-Decoder Design for Speech Enhancement [1.4037575966075835]
  1-D filters on raw audio are hard to train and often suffer from instabilities. We address these problems with hybrid solutions, combining theory-driven and data-driven approaches.
  arXiv  Detail & Related papers  (2024-08-30T15:49:31Z)
• FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection [61.9638234358049]
  FastFiD is a novel approach that executes sentence selection on encoded passages. This aids in retaining valuable sentences while reducing the context length required for generating answers.
  arXiv  Detail & Related papers  (2024-08-12T17:50:02Z)
• Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving [74.28510044056706]
  Existing methods usually adopt the decoupled encoder-decoder paradigm. In this work, we aim to alleviate the problem by two principles. We first predict a coarse-grained future position and action based on the encoder features. Then, conditioned on the position and action, the future scene is imagined to check the ramification if we drive accordingly.
  arXiv  Detail & Related papers  (2023-05-10T15:22:02Z)
• Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder [75.03283861464365]
  The seq2seq task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task.
  arXiv  Detail & Related papers  (2023-04-08T15:44:29Z)
• Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering [85.08146789409354]
  We propose a new architecture, Task-awaredomain for dense Retrieval (TASER) TASER enables parameter sharing by interleaving shared and specialized blocks in a single encoder. Our experiments show that TASER can achieve superior accuracy, surpassing BM25, while using about 60% of the parameters as bi-encoder dense retrievers.
  arXiv  Detail & Related papers  (2022-10-11T05:33:25Z)
• Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval [29.16807969384253]
  Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. We propose a framework to enhance the Dual-Encoders model with question answer cross-embeddings and a novel Geometry Alignment Mechanism (GAM) Our framework significantly improves Dual-Encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets.
  arXiv  Detail & Related papers  (2022-06-07T02:39:24Z)
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations [22.40667024030858]
  Bi-encoders produce fixed-dimensional sentence representations and are computationally efficient. Cross-encoders can leverage their attention heads to exploit inter-sentence interactions for better performance. Trans-Encoder combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi- and cross-encoders.
  arXiv  Detail & Related papers  (2021-09-27T14:06:47Z)
• Question Answering Infused Pre-training of General-Purpose Contextualized Representations [70.62967781515127]
  We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations. We accomplish this goal by training a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection.
  arXiv  Detail & Related papers  (2021-06-15T14:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.