Weakly Supervised Pre-Training for Multi-Hop Retriever

• URL: http://arxiv.org/abs/2106.09983v1
• Date: Fri, 18 Jun 2021 08:06:02 GMT
• Title: Weakly Supervised Pre-Training for Multi-Hop Retriever
• Authors: Yeon Seonwoo, Sang-Woo Lee, Ji-Hoon Kim, Jung-Woo Ha, Alice Oh
• Abstract summary: We propose a new method for weakly supervised multi-hop retriever pre-training without human efforts. Our method includes 1) a pre-training task for generating vector representations of complex questions, 2) a scalable data generation method that produces the nested structure of question and sub-question as weak supervision for pre-training, and 3) a pre-training model structure based on dense encoders.
• Score: 23.79574380039197
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In multi-hop QA, answering complex questions entails iterative document retrieval for finding the missing entity of the question. The main steps of this process are sub-question detection, document retrieval for the sub-question, and generation of a new query for the final document retrieval. However, building a dataset that contains complex questions with sub-questions and their corresponding documents requires costly human annotation. To address the issue, we propose a new method for weakly supervised multi-hop retriever pre-training without human efforts. Our method includes 1) a pre-training task for generating vector representations of complex questions, 2) a scalable data generation method that produces the nested structure of question and sub-question as weak supervision for pre-training, and 3) a pre-training model structure based on dense encoders. We conduct experiments to compare the performance of our pre-trained retriever with several state-of-the-art models on end-to-end multi-hop QA as well as document retrieval. The experimental results show that our pre-trained retriever is effective and also robust on limited data and computational resources.

Related papers

• Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering [45.82437926569949]
  Multi-Hop Question Answering tasks present a significant challenge for large language models. We introduce a novel generate-then-ground (GenGround) framework to solve a multi-hop question.
  arXiv  Detail & Related papers  (2024-06-21T06:26:38Z)
• Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization [89.04537372465612]
  Socratic pretraining is a question-driven, unsupervised pretraining objective designed to improve controllability in summarization tasks. Our results show that Socratic pretraining cuts task-specific labeled data requirements in half.
  arXiv  Detail & Related papers  (2022-12-20T17:27:10Z)
• Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer [80.50327229467993]
  We show that a single model trained end-to-end can achieve both competitive retrieval and QA performance. We show that end-to-end adaptation significantly boosts its performance on out-of-domain datasets in both supervised and unsupervised settings.
  arXiv  Detail & Related papers  (2022-12-05T04:51:21Z)
• UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph [89.98762327725112]
  Multi-hop Question Answering over Knowledge Graph(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question. We propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning.
  arXiv  Detail & Related papers  (2022-12-02T04:08:09Z)
• Questions Are All You Need to Train a Dense Passage Retriever [123.13872383489172]
  ART is a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. It uses a new document-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence documents, and (2) the documents are then used to compute the probability of reconstructing the original question.
  arXiv  Detail & Related papers  (2022-06-21T18:16:31Z)
• End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering [36.80395759543162]
  We present an end-to-end differentiable training method for retrieval-augmented open-domain question answering systems. We model retrieval decisions as latent variables over sets of relevant documents. Our proposed method outperforms all existing approaches of comparable size by 2-3% exact match points.
  arXiv  Detail & Related papers  (2021-06-09T19:25:37Z)
• Distilling Knowledge from Reader to Retriever for Question Answering [16.942581590186343]
  We propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation. We evaluate our method on question answering, obtaining state-of-the-art results.
  arXiv  Detail & Related papers  (2020-12-08T17:36:34Z)
• Open Question Answering over Tables and Text [55.8412170633547]
  In open question answering (QA), the answer to a question is produced by retrieving and then analyzing documents that might contain answers to the question. Most open QA systems have considered only retrieving information from unstructured text. We present a new large-scale dataset Open Table-and-Text Question Answering (OTT-QA) to evaluate performance on this task.
  arXiv  Detail & Related papers  (2020-10-20T16:48:14Z)
• Pre-training Tasks for Embedding-based Large-scale Retrieval [68.01167604281578]
  We consider the large-scale query-document retrieval problem. Given a query (e.g., a question), return the set of relevant documents from a large document corpus. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks.
  arXiv  Detail & Related papers  (2020-02-10T16:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.