RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open
Domain Question Answering
- URL: http://arxiv.org/abs/2010.10757v1
- Date: Wed, 21 Oct 2020 04:28:42 GMT
- Title: RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open
Domain Question Answering
- Authors: Srinivasan Iyer, Sewon Min, Yashar Mehdad, Wen-tau Yih
- Abstract summary: We develop a simple and effective re-ranking approach (RECONSIDER) for span-extraction tasks.
RECONSIDER is trained on positive and negative examples extracted from high confidence predictions of MRC models.
It uses in-passage span annotations to perform span-focused re-ranking over a smaller candidate set.
- Score: 49.024513062811685
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain
Question Answering (QA) are typically trained for span selection using
distantly supervised positive examples and heuristically retrieved negative
examples. This training scheme possibly explains empirical observations that
these models achieve a high recall amongst their top few predictions, but a low
overall accuracy, motivating the need for answer re-ranking. We develop a
simple and effective re-ranking approach (RECONSIDER) for span-extraction
tasks, that improves upon the performance of large pre-trained MRC models.
RECONSIDER is trained on positive and negative examples extracted from high
confidence predictions of MRC models, and uses in-passage span annotations to
perform span-focused re-ranking over a smaller candidate set. As a result,
RECONSIDER learns to eliminate close false positive passages, and achieves a
new state of the art on four QA tasks, including 45.5% Exact Match accuracy on
Natural Questions with real user questions, and 61.7% on TriviaQA.
Related papers
- Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple
Logits Retargeting Approach [102.0769560460338]
We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class.
Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
arXiv Detail & Related papers (2024-03-01T03:27:08Z) - Reframing Offline Reinforcement Learning as a Regression Problem [0.0]
The study proposes the reformulation of offline reinforcement learning as a regression problem that can be solved with decision trees.
We observe that with gradient-boosted trees, the agent training and inference are very fast.
Despite the simplification inherent in this reformulated problem, our agent demonstrates performance that is at least on par with established methods.
arXiv Detail & Related papers (2024-01-21T23:50:46Z) - Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice.
We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
arXiv Detail & Related papers (2023-12-27T09:03:43Z) - Augmenting Unsupervised Reinforcement Learning with Self-Reference [63.68018737038331]
Humans possess the ability to draw on past experiences explicitly when learning new tasks.
We propose the Self-Reference (SR) approach, an add-on module explicitly designed to leverage historical information.
Our approach achieves state-of-the-art results in terms of Interquartile Mean (IQM) performance and Optimality Gap reduction on the Unsupervised Reinforcement Learning Benchmark.
arXiv Detail & Related papers (2023-11-16T09:07:34Z) - Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large
Language Models [37.016804744883096]
We introduce Contrastive Novelty-Augmented Learning (CoNAL), a two-step method that generates OOD examples representative of novel classes, then trains to decrease confidence on them.
When trained with CoNAL, classifiers improve in their ability to detect and abstain on novel class examples over prior methods by an average of 2.3% in terms of accuracy.
arXiv Detail & Related papers (2022-11-28T19:03:35Z) - Towards Robust Visual Question Answering: Making the Most of Biased
Samples via Contrastive Learning [54.61762276179205]
We propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples.
Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples.
We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.
arXiv Detail & Related papers (2022-10-10T11:05:21Z) - Improving Passage Retrieval with Zero-Shot Question Generation [109.11542468380331]
We propose a simple and effective re-ranking method for improving passage retrieval in open question answering.
The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage.
arXiv Detail & Related papers (2022-04-15T14:51:41Z) - Towards Confident Machine Reading Comprehension [7.989756186727329]
We propose a novel post-prediction confidence estimation model, which we call Mr.C (short for Mr. Confident)
Mr.C can be trained to improve a system's ability to refrain from making incorrect predictions with improvements of up to 4 points as measured by Area Under the Curve (AUC) scores.
arXiv Detail & Related papers (2021-01-20T03:02:12Z) - End-to-End Training of Neural Retrievers for Open-Domain Question
Answering [32.747113232867825]
It remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers.
We propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans.
We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
arXiv Detail & Related papers (2021-01-02T09:05:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.