Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval

• URL: http://arxiv.org/abs/2206.02978v1
• Date: Tue, 7 Jun 2022 02:39:24 GMT
• Title: Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval
• Authors: Yanmeng Wang, Jun Bai, Ye Wang, Jianfei Zhang, Wenge Rong, Zongcheng Ji, Shaojun Wang, Jing Xiao
• Abstract summary: Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. We propose a framework to enhance the Dual-Encoders model with question answer cross-embeddings and a novel Geometry Alignment Mechanism (GAM) Our framework significantly improves Dual-Encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets.
• Score: 29.16807969384253
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. Currently most conventional Dual-Encoders learn the semantic representations of questions and answers merely through matching score. Researchers proposed to introduce the QA interaction features in scoring function but at the cost of low efficiency in inference stage. To keep independent encoding of questions and answers during inference stage, variational auto-encoder is further introduced to reconstruct answers (questions) from question (answer) embeddings as an auxiliary task to enhance QA interaction in representation learning in training stage. However, the needs of text generation and answer retrieval are different, which leads to hardness in training. In this work, we propose a framework to enhance the Dual-Encoders model with question answer cross-embeddings and a novel Geometry Alignment Mechanism (GAM) to align the geometry of embeddings from Dual-Encoders with that from Cross-Encoders. Extensive experimental results show that our framework significantly improves Dual-Encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets.

Related papers

• Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering [2.98667511228225]
  ReRe is an encoder-decoder architecture model using a pre-trained clip vision encoder and a pre-trained GPT-2 language model as a decoder. ReRe outperforms previous methods in VQA accuracy and explanation score and shows improvement in NLE with more persuasive, reliability.
  arXiv  Detail & Related papers  (2024-08-30T04:39:43Z)
• Exploring Diverse Methods in Visual Question Answering [0.6707149143800017]
  This study explores innovative methods for improving Visual Question Answering (VQA) using Generative Adversarial Networks (GANs), autoencoders, and attention mechanisms. GAN-based approaches aim to generate answer embeddings conditioned on image and question inputs, showing potential but struggling with more complex tasks. Autoencoder-based techniques focus on learning optimal embeddings for questions and images, achieving comparable results with GAN due to better ability on complex questions.
  arXiv  Detail & Related papers  (2024-04-21T07:34:44Z)
• Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders [89.29256833403169]
  We introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods. KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.
  arXiv  Detail & Related papers  (2023-03-31T15:44:13Z)
• Modeling Multi-hop Question Answering as Single Sequence Prediction [88.72621430714985]
  We propose a simple generative approach (PathFid) that extends the task beyond just answer generation. PathFid explicitly models the reasoning process to resolve the answer for multi-hop questions. Our experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets.
  arXiv  Detail & Related papers  (2022-05-18T21:57:59Z)
• Exploring Dual Encoder Architectures for Question Answering [17.59582094233306]
  Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. There are two major types of dual encoders, Siamese Duals (SDE) and Asymmetric Dual architectures (ADE)
  arXiv  Detail & Related papers  (2022-04-14T17:21:14Z)
• Double Retrieval and Ranking for Accurate Question Answering [120.69820139008138]
  We show that an answer verification step introduced in Transformer-based answer selection models can significantly improve the state of the art in Question Answering. The results on three well-known datasets for AS2 show consistent and significant improvement of the state of the art.
  arXiv  Detail & Related papers  (2022-01-16T06:20:07Z)
• Distilled Dual-Encoder Model for Vision-Language Understanding [50.42062182895373]
  We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks. We show that applying the cross-modal attention distillation for both pre-training and fine-tuning stages achieves further improvements.
  arXiv  Detail & Related papers  (2021-12-16T09:21:18Z)
• Hybrid Autoregressive Solver for Scalable Abductive Natural Language Inference [2.867517731896504]
  We propose a hybrid abductive solver that autoregressively combines a dense bi-encoder with a sparse model of explanatory power. Our experiments demonstrate that the proposed framework can achieve performance comparable with the state-of-the-art cross-encoder.
  arXiv  Detail & Related papers  (2021-07-25T19:29:53Z)
• Question Answering Infused Pre-training of General-Purpose Contextualized Representations [70.62967781515127]
  We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations. We accomplish this goal by training a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection.
  arXiv  Detail & Related papers  (2021-06-15T14:45:15Z)
• Crossing Variational Autoencoders for Answer Retrieval [50.17311961755684]
  Question-answer alignment and question/answer semantics are two important signals for learning the representations. We propose to cross variational auto-encoders by generating questions with aligned answers and generating answers with aligned questions.
  arXiv  Detail & Related papers  (2020-05-06T01:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.