QA-NatVer: Question Answering for Natural Logic-based Fact Verification
- URL: http://arxiv.org/abs/2310.14198v1
- Date: Sun, 22 Oct 2023 06:27:31 GMT
- Title: QA-NatVer: Question Answering for Natural Logic-based Fact Verification
- Authors: Rami Aly and Marek Strong and Andreas Vlachos
- Abstract summary: We propose to use question answering to predict natural logic operators.
In a few-shot setting on FEVER, our approach outperforms the best baseline by $4.3$ accuracy points.
A human evaluation indicates that our approach produces more plausible with fewer erroneous natural logic operators than previous natural logic-based systems.
- Score: 11.002475880349452
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Fact verification systems assess a claim's veracity based on evidence. An
important consideration in designing them is faithfulness, i.e. generating
explanations that accurately reflect the reasoning of the model. Recent works
have focused on natural logic, which operates directly on natural language by
capturing the semantic relation of spans between an aligned claim with its
evidence via set-theoretic operators. However, these approaches rely on
substantial resources for training, which are only available for high-resource
languages. To this end, we propose to use question answering to predict natural
logic operators, taking advantage of the generalization capabilities of
instruction-tuned language models. Thus, we obviate the need for annotated
training data while still relying on a deterministic inference system. In a
few-shot setting on FEVER, our approach outperforms the best baseline by $4.3$
accuracy points, including a state-of-the-art pre-trained seq2seq natural logic
system, as well as a state-of-the-art prompt-based classifier. Our system
demonstrates its robustness and portability, achieving competitive performance
on a counterfactual dataset and surpassing all approaches without further
annotation on a Danish verification dataset. A human evaluation indicates that
our approach produces more plausible proofs with fewer erroneous natural logic
operators than previous natural logic-based systems.
Related papers
- TabVer: Tabular Fact Verification with Natural Logic [11.002475880349452]
We propose a set-theoretic interpretation of numerals and arithmetic functions in the context of natural logic.
We leverage large language models to generate arithmetic expressions by generating questions about salient parts of a claim which are answered by executing functions on tables.
In a few-shot setting on FEVEROUS, we achieve an accuracy of 71.4, outperforming both fully neural and symbolic reasoning models by 3.4 points.
arXiv Detail & Related papers (2024-11-02T00:36:34Z) - Zero-Shot Fact Verification via Natural Logic and Large Language Models [9.789552436902342]
We propose a zero-shot method that utilizes the generalization capabilities of instruction-tuned large language models.
Our method achieves an average accuracy improvement of 8.96 points over the best-performing baseline.
Current systems trained on natural logic data do not generalize well to other domains.
arXiv Detail & Related papers (2024-10-04T11:57:32Z) - Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars [0.6537995248511139]
We present a declarative framework with flexible context-sensitive rules binding multiple languages.
We construct first-order logic problems by selecting up to 32 premises and one hypothesis.
We demonstrate that using semantic constraints during generation and careful English verbalization of predicates enhances logical reasoning without hurting natural English tasks.
arXiv Detail & Related papers (2024-06-16T18:10:49Z) - A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains [33.46649770312231]
Prompting language models to provide step-by-step answers is a prominent approach for complex reasoning tasks.
No fine-grained step-level datasets are available to enable thorough evaluation of such verification methods.
We introduce REVEAL: Reasoning Verification Evaluation, a dataset to benchmark automatic verifiers of complex Chain-of-Thought reasoning.
arXiv Detail & Related papers (2024-02-01T12:46:45Z) - Empower Nested Boolean Logic via Self-Supervised Curriculum Learning [67.46052028752327]
We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested logic.
To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method textitCurriculum Logical Reasoning (textscClr)
arXiv Detail & Related papers (2023-10-09T06:54:02Z) - Modeling Hierarchical Reasoning Chains by Linking Discourse Units and
Key Phrases for Reading Comprehension [80.99865844249106]
We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning.
Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
arXiv Detail & Related papers (2023-06-21T07:34:27Z) - ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness [67.49087159888298]
ReCEval is a framework that evaluates reasoning chains via two key properties: correctness and informativeness.
We show that ReCEval effectively identifies various error types and yields notable improvements compared to prior methods.
arXiv Detail & Related papers (2023-04-21T02:19:06Z) - APOLLO: A Simple Approach for Adaptive Pretraining of Language Models
for Logical Reasoning [73.3035118224719]
We propose APOLLO, an adaptively pretrained language model that has improved logical reasoning abilities.
APOLLO performs comparably on ReClor and outperforms baselines on LogiQA.
arXiv Detail & Related papers (2022-12-19T07:40:02Z) - Logical Satisfiability of Counterfactuals for Faithful Explanations in
NLI [60.142926537264714]
We introduce the methodology of Faithfulness-through-Counterfactuals.
It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation.
It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
arXiv Detail & Related papers (2022-05-25T03:40:59Z) - Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason
Over Implicit Knowledge [96.92252296244233]
Large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control.
We show that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements.
Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
arXiv Detail & Related papers (2020-06-11T17:02:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.