Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes

• URL: http://arxiv.org/abs/2204.07693v1
• Date: Sat, 16 Apr 2022 01:03:36 GMT
• Title: Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes
• Authors: Kaige Xie, Sarah Wiegreffe, Mark Riedl
• Abstract summary: Multi-hop Question Answering (QA) is a challenging task since it requires an accurate aggregation of information from multiple context paragraphs. Recent work in multi-hop QA has shown that performance can be boosted by first decomposing the questions into simpler, single-hop questions. We show that decomposition is an effective form of probing QA systems as well as a promising approach to explanation generation.
• Score: 14.302797773412543
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-hop Question Answering (QA) is a challenging task since it requires an accurate aggregation of information from multiple context paragraphs and a thorough understanding of the underlying reasoning chains. Recent work in multi-hop QA has shown that performance can be boosted by first decomposing the questions into simpler, single-hop questions. In this paper, we explore one additional utility of the multi-hop decomposition from the perspective of explainable NLP: to create explanation by probing a neural QA model with them. We hypothesize that in doing so, users will be better able to construct a mental model of when the underlying QA system will give the correct answer. Through human participant studies, we verify that exposing the decomposition probes and answers to the probes to users can increase their ability to predict system performance on a question instance basis. We show that decomposition is an effective form of probing QA systems as well as a promising approach to explanation generation. In-depth analyses show the need for improvements in decomposition systems.

Related papers

• Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering [85.79940770146557]
  We decompose multi-hop questions into multiple corresponding single-hop questions. We find marked inconsistency in QA models' answers on these pairs of ostensibly identical question chains. When trained only on single-hop questions, models generalize poorly to multi-hop questions.
  arXiv  Detail & Related papers  (2022-10-09T11:48:07Z)
• Locate Then Ask: Interpretable Stepwise Reasoning for Multi-hop Question Answering [71.49131159045811]
  Multi-hop reasoning requires aggregating multiple documents to answer a complex question. Existing methods usually decompose the multi-hop question into simpler single-hop questions. We propose an interpretable stepwise reasoning framework to incorporate both single-hop supporting sentence identification and single-hop question generation.
  arXiv  Detail & Related papers  (2022-08-22T13:24:25Z)
• Interpretable AMR-Based Question Decomposition for Multi-hop Question Answering [12.35571328854374]
  We propose a Question Decomposition method based on Abstract Meaning Representation (QDAMR) for multi-hop QA. We decompose a multi-hop question into simpler sub-questions and answer them in order. Experimental results on HotpotQA demonstrate that our approach is competitive for interpretable reasoning.
  arXiv  Detail & Related papers  (2022-06-16T23:46:33Z)
• Modeling Multi-hop Question Answering as Single Sequence Prediction [88.72621430714985]
  We propose a simple generative approach (PathFid) that extends the task beyond just answer generation. PathFid explicitly models the reasoning process to resolve the answer for multi-hop questions. Our experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets.
  arXiv  Detail & Related papers  (2022-05-18T21:57:59Z)
• Ask to Understand: Question Generation for Multi-hop Question Answering [11.626390908264872]
  Multi-hop Question Answering (QA) requires the machine to answer complex questions by finding scattering clues and reasoning from multiple documents. We propose a novel method to complete multi-hop QA from the perspective of Question Generation (QG)
  arXiv  Detail & Related papers  (2022-03-17T04:02:29Z)
• Multi-hop Question Generation with Graph Convolutional Network [58.31752179830959]
  Multi-hop Question Generation (QG) aims to generate answer-related questions by aggregating and reasoning over multiple scattered evidence from different paragraphs. We propose Multi-Hop volution Fusion Network for Question Generation (MulQG), which does context encoding in multiple hops. Our proposed model is able to generate fluent questions with high completeness and outperforms the strongest baseline by 20.8% in the multi-hop evaluation.
  arXiv  Detail & Related papers  (2020-10-19T06:15:36Z)
• Reinforced Multi-task Approach for Multi-hop Question Generation [47.15108724294234]
  We take up Multi-hop question generation, which aims at generating relevant questions based on supporting facts in the context. We employ multitask learning with the auxiliary task of answer-aware supporting fact prediction to guide the question generator. We demonstrate the effectiveness of our approach through experiments on the multi-hop question answering dataset, HotPotQA.
  arXiv  Detail & Related papers  (2020-04-05T10:16:59Z)
• Unsupervised Question Decomposition for Question Answering [102.56966847404287]
  We propose an algorithm for One-to-N Unsupervised Sequence Sequence (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets.
  arXiv  Detail & Related papers  (2020-02-22T19:40:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.