Machine Reading Comprehension using Case-based Reasoning
- URL: http://arxiv.org/abs/2305.14815v4
- Date: Tue, 5 Dec 2023 20:59:58 GMT
- Title: Machine Reading Comprehension using Case-based Reasoning
- Authors: Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das,
Manzil Zaheer, Jay-Yoon Lee, Hannaneh Hajishirzi, Andrew McCallum
- Abstract summary: We present an accurate and interpretable method for answer extraction in machine reading comprehension.
Our method builds upon the hypothesis that contextualized answers to similar questions share semantic similarities with each other.
- Score: 92.51061570746077
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We present an accurate and interpretable method for answer extraction in
machine reading comprehension that is reminiscent of case-based reasoning (CBR)
from classical AI. Our method (CBR-MRC) builds upon the hypothesis that
contextualized answers to similar questions share semantic similarities with
each other. Given a test question, CBR-MRC first retrieves a set of similar
cases from a nonparametric memory and then predicts an answer by selecting the
span in the test context that is most similar to the contextualized
representations of answers in the retrieved cases. The semi-parametric nature
of our approach allows it to attribute a prediction to the specific set of
evidence cases, making it a desirable choice for building reliable and
debuggable QA systems. We show that CBR-MRC provides high accuracy comparable
with large reader models and outperforms baselines by 11.5 and 8.4 EM on
NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability
of CBR-MRC in identifying not just the correct answer tokens but also the span
with the most relevant supporting evidence. Lastly, we observe that contexts
for certain question types show higher lexical diversity than others and find
that CBR-MRC is robust to these variations while performance using
fully-parametric methods drops.
Related papers
- QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answering [32.436530949623155]
We propose a unique scaling strategy to capture latent semantic center features of queries.
These features are seamlessly integrated into traditional query and passage embeddings.
Our approach diminishes sensitivity to variations in text format and boosts the model's capability in pinpointing accurate answers.
arXiv Detail & Related papers (2024-04-30T07:34:42Z) - CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering [1.0760413363405308]
Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input.
Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM.
We introduce CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases.
arXiv Detail & Related papers (2024-04-04T21:47:43Z) - Momentum Contrastive Pre-training for Question Answering [54.57078061878619]
MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs.
Our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.
arXiv Detail & Related papers (2022-12-12T08:28:22Z) - Benchmarking Answer Verification Methods for Question Answering-Based
Summarization Evaluation Metrics [74.28810048824519]
Question answering-based summarization evaluation metrics must automatically determine whether the QA model's prediction is correct or not.
We benchmark the lexical answer verification methods which have been used by current QA-based metrics as well as two more sophisticated text comparison methods.
arXiv Detail & Related papers (2022-04-21T15:43:45Z) - Knowledge Base Question Answering by Case-based Reasoning over Subgraphs [81.22050011503933]
We show that our model answers queries requiring complex reasoning patterns more effectively than existing KG completion algorithms.
The proposed model outperforms or performs competitively with state-of-the-art models on several KBQA benchmarks.
arXiv Detail & Related papers (2022-02-22T01:34:35Z) - Case-based Reasoning for Natural Language Queries over Knowledge Bases [41.54465521439727]
We propose a neuro-symbolic CBR approach for question answering over large knowledge bases.
CBR-KBQA consists of two modules: a non-parametric memory that stores cases and a parametric model.
We show that CBR-KBQA can effectively derive novel combination of relations not presented in case memory.
arXiv Detail & Related papers (2021-04-18T07:50:31Z) - Counterfactual Variable Control for Robust and Interpretable Question
Answering [57.25261576239862]
Deep neural network based question answering (QA) models are neither robust nor explainable in many cases.
In this paper, we inspect such spurious "capability" of QA models using causal inference.
We propose a novel approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlation.
arXiv Detail & Related papers (2020-10-12T10:09:05Z) - Robust Question Answering Through Sub-part Alignment [53.94003466761305]
We model question answering as an alignment problem.
We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets.
arXiv Detail & Related papers (2020-04-30T09:10:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.