R3: A Reading Comprehension Benchmark Requiring Reasoning Processes
- URL: http://arxiv.org/abs/2004.01251v1
- Date: Thu, 2 Apr 2020 20:39:12 GMT
- Title: R3: A Reading Comprehension Benchmark Requiring Reasoning Processes
- Authors: Ran Wang, Kun Tao, Dingjie Song, Zhilong Zhang, Xiao Ma, Xi'ao Su,
Xinyu Dai
- Abstract summary: We introduce a formalism for reasoning over unstructured text, namely Text Reasoning Representation Meaning (TRMR)
TRMR consists of three phrases, which is expressive enough to characterize the reasoning process to answer reading comprehension questions.
We release the R3 dataset, a textbfReading comprehension benchmark textbfRequiring textbfReasoning processes.
- Score: 23.320171155581175
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Existing question answering systems can only predict answers without explicit
reasoning processes, which hinder their explainability and make us overestimate
their ability of understanding and reasoning over natural language. In this
work, we propose a novel task of reading comprehension, in which a model is
required to provide final answers and reasoning processes. To this end, we
introduce a formalism for reasoning over unstructured text, namely Text
Reasoning Meaning Representation (TRMR). TRMR consists of three phrases, which
is expressive enough to characterize the reasoning process to answer reading
comprehension questions. We develop an annotation platform to facilitate TRMR's
annotation, and release the R3 dataset, a \textbf{R}eading comprehension
benchmark \textbf{R}equiring \textbf{R}easoning processes. R3 contains over 60K
pairs of question-answer pairs and their TRMRs. Our dataset is available at:
\url{http://anonymous}.
Related papers
- R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning [62.742230250513025]
Retrieval-Augmented Generation (RAG) integrates external knowledge with Large Language Models (LLMs) to enhance factual correctness and hallucination.<n>We propose $textbfR3-RAG$, which uses $textbfR$einforcement learning to make the LLM learn how to $textbfR$eason and $textbfR$etrieve step by step, thus retrieving comprehensive external knowledge and leading to correct answers.
arXiv Detail & Related papers (2025-05-26T12:25:37Z) - Identification of Entailment and Contradiction Relations between Natural Language Sentences: A Neurosymbolic Approach [8.931767102433637]
Natural language inference (NLI) is an important aspect of natural language understanding.
To address the need for an explainable approach to RTE, we propose a novel pipeline that is based on translating text into an abstract representation.
We then translate the AMR graph into propositional logic and use a SAT solver for automated reasoning.
arXiv Detail & Related papers (2024-05-02T13:06:24Z) - Widely Interpretable Semantic Representation: Frameless Meaning
Representation for Broader Applicability [10.710058244695128]
This paper presents a novel semantic representation, WISeR, that overcomes challenges for Abstract Meaning Representation (AMR)
Despite its strengths, AMR is not easily applied to languages or domains without predefined semantic frames.
We create a new corpus of 1K English dialogue sentences in both WISeR and AMR WISeR.
arXiv Detail & Related papers (2023-09-12T17:44:40Z) - S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical
Learning [46.787034512390434]
VQA Natural Language Explanation (VQA-NLE) task aims to explain the decision-making process of VQA models in natural language.
We propose a new Semi-Supervised VQA-NLE via Self-Critical Learning (S3C)
S3C evaluates the candidate explanations by answering rewards to improve the logical consistency between answers and rationales.
arXiv Detail & Related papers (2023-09-05T11:47:51Z) - STREET: A Multi-Task Structured Reasoning and Explanation Benchmark [56.555662318619135]
We introduce a unified multi-task and multi-domain natural language reasoning and explanation benchmark.
We expect models to not only answer questions, but also produce step-by-step structured explanations describing how premises in the question are used to produce intermediate conclusions that can prove the correctness of a certain answer.
arXiv Detail & Related papers (2023-02-13T22:34:02Z) - Successive Prompting for Decomposing Complex Questions [50.00659445976735]
Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting.
We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution.
Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
arXiv Detail & Related papers (2022-12-08T06:03:38Z) - The Unreliability of Explanations in Few-Shot In-Context Learning [50.77996380021221]
We focus on two NLP tasks that involve reasoning over text, namely question answering and natural language inference.
We show that explanations judged as good by humans--those that are logically consistent with the input--usually indicate more accurate predictions.
We present a framework for calibrating model predictions based on the reliability of the explanations.
arXiv Detail & Related papers (2022-05-06T17:57:58Z) - Dynamic Semantic Graph Construction and Reasoning for Explainable
Multi-hop Science Question Answering [50.546622625151926]
We propose a new framework to exploit more valid facts while obtaining explainability for multi-hop QA.
Our framework contains three new ideas: (a) tt AMR-SG, an AMR-based Semantic Graph, constructed by candidate fact AMRs to uncover any hop relations among question, answer and multiple facts, (b) a novel path-based fact analytics approach exploiting tt AMR-SG to extract active facts from a large fact pool to answer questions, and (c) a fact-level relation modeling leveraging graph convolution network (GCN) to guide the reasoning process.
arXiv Detail & Related papers (2021-05-25T09:14:55Z) - Discrete Reasoning Templates for Natural Language Understanding [79.07883990966077]
We present an approach that reasons about complex questions by decomposing them to simpler subquestions.
We derive the final answer according to instructions in a predefined reasoning template.
We show that our approach is competitive with the state-of-the-art while being interpretable and requires little supervision.
arXiv Detail & Related papers (2021-04-05T18:56:56Z) - Break It Down: A Question Understanding Benchmark [79.41678884521801]
We introduce a Question Decomposition Representation Meaning (QDMR) for questions.
QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question.
We release the Break dataset, containing over 83K pairs of questions and their QDMRs.
arXiv Detail & Related papers (2020-01-31T11:04:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.