Related papers: ExpMRC: Explainability Evaluation for Machine Reading Comprehension

ExpMRC: Explainability Evaluation for Machine Reading Comprehension

URL: http://arxiv.org/abs/2105.04126v1
Date: Mon, 10 May 2021 06:00:20 GMT
Title: ExpMRC: Explainability Evaluation for Machine Reading Comprehension
Authors: Yiming Cui, Ting Liu, Wanxiang Che, Zhigang Chen, Shijin Wang
Abstract summary: We propose a new benchmark called ExpMRC for evaluating the explainability of the Machine Reading systems. We use state-of-the-art pre-trained language models to build baseline systems and adopt various unsupervised approaches to extract evidence without a human-annotated training set.
Score: 42.483940360860096
License: http://creativecommons.org/licenses/by-sa/4.0/
Abstract: Achieving human-level performance on some of Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE$^+$, and C$^3$ with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art pre-trained language models to build baseline systems and adopt various unsupervised approaches to extract evidence without a human-annotated training set. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources will be available through https://github.com/ymcui/expmrc

Related papers

From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader [130.45769668885487]
Pre-trained Machine Reader (PMR) is a novel method for retrofitting masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data. To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data. PMR has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation.
arXiv Detail & Related papers (2022-12-09T10:21:56Z)
Feeding What You Need by Understanding What You Learned [54.400455868448695]
Machine Reading (MRC) reveals the ability to understand a given text passage and answer questions based on it. Existing research works in MRC rely heavily on large-size models and corpus to improve the performance evaluated by metrics such as Exact Match. We argue that a deep understanding of model capabilities and data properties can help us feed a model with appropriate training data.
arXiv Detail & Related papers (2022-03-05T14:15:59Z)
Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision [34.01738910736325]
We propose a new framework for unsupervised machine reading comprehension (MRC) We learn to spot answer spans in documents via self-supervised learning, by designing a self-supervision pretext task for MRC - Spotting-MLM. Experiments show that our method achieves a new state-of-the-art performance for unsupervised MRC.
arXiv Detail & Related papers (2021-07-19T02:14:36Z)
REPT: Bridging Language Models and Machine Reading Comprehensionvia Retrieval-Based Pre-training [45.21249008835556]
We present REPT, a REtrieval-based Pre-Training approach to bridge the gap between general PLMs and MRC. In particular, we introduce two self-supervised tasks to strengthen evidence extraction during pre-training. Our approach is able to enhance the capacity of evidence extraction without explicit supervision.
arXiv Detail & Related papers (2021-05-10T08:54:46Z)
Coreference Reasoning in Machine Reading Comprehension [100.75624364257429]
We show that coreference reasoning in machine reading comprehension is a greater challenge than was earlier thought. We propose a methodology for creating reading comprehension datasets that better reflect the challenges of coreference reasoning. This allows us to show an improvement in the reasoning abilities of state-of-the-art models across various MRC datasets.
arXiv Detail & Related papers (2020-12-31T12:18:41Z)
A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets [5.54205518616467]
Machine Reading (MRC) is a challenging Natural Language Processing(NLP) research field with wide real-world applications. A lot of MRC models have already surpassed human performance on various benchmark datasets. This shows the need for improving existing datasets, evaluation metrics, and models to move current MRC models toward "real" understanding.
arXiv Detail & Related papers (2020-06-21T19:18:54Z)
A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction [89.88061141170512]
We present a Self-Training method (STM) to train machine reading comprehension models. At each iteration, a base MRC model is trained with golden answers and noisy evidence labels. The trained model will predict pseudo evidence labels as extra supervision in the next iteration.
arXiv Detail & Related papers (2020-05-11T15:26:07Z)
Retrospective Reader for Machine Reading Comprehension [90.6069071495214]
Machine reading comprehension (MRC) is an AI challenge that requires machine to determine the correct answers to questions based on a given passage. When unanswerable questions are involved in the MRC task, an essential verification module called verifier is especially required in addition to the encoder. This paper devotes itself to exploring better verifier design for the MRC task with unanswerable questions.
arXiv Detail & Related papers (2020-01-27T11:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.