World Knowledge in Multiple Choice Reading Comprehension

• URL: http://arxiv.org/abs/2211.07040v2
• Date: Tue, 30 May 2023 12:37:15 GMT
• Title: World Knowledge in Multiple Choice Reading Comprehension
• Abstract summary: Multiple choice reading comprehension systems are able to answer questions significantly better than random on average without access to the contextual passage. This paper examines the possibility of exploiting this observation as a tool for test designers to ensure that the use of "world knowledge" is acceptable for a particular set of questions. Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently it has been shown that without any access to the contextual passage, multiple choice reading comprehension (MCRC) systems are able to answer questions significantly better than random on average. These systems use their accumulated "world knowledge" to directly answer questions, rather than using information from the passage. This paper examines the possibility of exploiting this observation as a tool for test designers to ensure that the use of "world knowledge" is acceptable for a particular set of questions. We propose information-theory based metrics that enable the level of "world knowledge" exploited by systems to be assessed. Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question. We demonstrate that questions with low expected number of options, and hence answerable by the shortcut system, are often similarly answerable by humans without context. This highlights that the general knowledge 'shortcuts' could be equally used by exam candidates, and that our proposed metrics may be helpful for future test designers to monitor the quality of questions.

Related papers

• KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education? [41.49674849980441]
  Retrieval-Augmented Generation (RAG) systems show remarkable potential as question answering tools in the K-12 Education domain.<n> discrepancies between these textbooks and the parametric knowledge inherent in Large Language Models (LLMs) can undermine the effectiveness of RAG systems.<n>KnowShiftQA simulates these discrepancies by applying deliberate hypothetical knowledge updates to both answers and source documents.
  arXiv  Detail & Related papers  (2024-12-12T06:38:40Z)
• How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading [60.19226384241482]
  We introduce GuidingQ, a dataset of 10K in-text questions from textbooks and scientific articles. We explore various approaches to generate such questions using language models. We conduct a human study to understand the implication of such questions on reading comprehension.
  arXiv  Detail & Related papers  (2024-07-19T13:42:56Z)
• Analyzing Multiple-Choice Reading and Listening Comprehension Tests [0.0]
  This work investigates how much of a contextual passage needs to be read in multiple-choice reading based on conversation transcriptions and listening comprehension tests to be able to work out the correct answer. We find that automated reading comprehension systems can perform significantly better than random with partial or even no access to the context passage.
  arXiv  Detail & Related papers  (2023-07-03T14:55:02Z)
• Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision [53.692793122749414]
  We introduce a medical question understanding and answering system with knowledge grounding and semantic self-supervision. Our system is a pipeline that first summarizes a long, medical, user-written question, using a supervised summarization loss. The system first matches the summarized user question with an FAQ from a trusted medical knowledge base, and then retrieves a fixed number of relevant sentences from the corresponding answer document.
  arXiv  Detail & Related papers  (2022-09-30T08:20:32Z)
• Multiple-Choice Question Generation: Towards an Automated Assessment Framework [0.0]
  transformer-based pretrained language models have demonstrated the ability to produce appropriate questions from a context paragraph. We focus on a fully automated multiple-choice question generation (MCQG) system where both the question and possible answers must be generated from the context paragraph.
  arXiv  Detail & Related papers  (2022-09-23T19:51:46Z)
• Interactive Question Answering Systems: Literature Review [17.033640293433397]
  Interactive question answering is a recently proposed and increasingly popular solution that resides at the intersection of question answering and dialogue systems. By permitting the user to ask more questions, interactive question answering enables users to dynamically interact with the system and receive more precise results. This survey offers a detailed overview of the interactive question-answering methods that are prevalent in current literature.
  arXiv  Detail & Related papers  (2022-09-04T13:46:54Z)
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge [77.77684299758494]
  We extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG) First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network (KNN), which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. To handle the diversity and complexity of KG information, we propose a Contextualized Knowledge-aware Attentive Neural Network (CKANN), which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network (GCN) and comprehensively learns context-based and knowledge-based sentence representation via
  arXiv  Detail & Related papers  (2021-04-12T05:52:20Z)
• CBench: Towards Better Evaluation of Question Answering Over Knowledge Graphs [3.631024220680066]
  We introduce CBench, an informative benchmarking suite for analyzing benchmarks and evaluating question answering systems. CBench can be used to analyze existing benchmarks with respect to several fine-grained linguistic, syntactic, and structural properties of the questions and queries.
  arXiv  Detail & Related papers  (2021-04-05T15:41:14Z)
• KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA [107.7091094498848]
  One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pre-training and supervised training data with transformer-based models.
  arXiv  Detail & Related papers  (2020-12-20T20:13:02Z)
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding [140.5911760063681]
  We propose a novel dataset named Knowledge-Routed Visual Question Reasoning for VQA model evaluation. We generate the question-answer pair based on both the Visual Genome scene graph and an external knowledge base with controlled programs.
  arXiv  Detail & Related papers  (2020-12-14T00:33:44Z)
• Difference-aware Knowledge Selection for Knowledge-grounded Conversation Generation [101.48602006200409]
  We propose a difference-aware knowledge selection method for multi-turn knowledge-grounded dialogs. It first computes the difference between the candidate knowledge sentences provided at the current turn and those chosen in the previous turns. Then, the differential information is fused with or disentangled from the contextual information to facilitate final knowledge selection.
  arXiv  Detail & Related papers  (2020-09-20T07:47:26Z)
• Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations [41.74162467619795]
  We investigate the problem of generating informative questions in information-asymmetric conversations. To generate pragmatic questions, we use reinforcement learning to optimize an informativeness metric. We demonstrate that the resulting pragmatic questioner substantially improves the informativeness and specificity of questions generated over a baseline model.
  arXiv  Detail & Related papers  (2020-04-30T00:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.