RobustLR: Evaluating Robustness to Logical Perturbation in Deductive
Reasoning
- URL: http://arxiv.org/abs/2205.12598v1
- Date: Wed, 25 May 2022 09:23:50 GMT
- Title: RobustLR: Evaluating Robustness to Logical Perturbation in Deductive
Reasoning
- Authors: Soumya Sanyal, Zeyi Liao, Xiang Ren
- Abstract summary: Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in English natural language.
We propose RobustLR to evaluate the robustness of these models to minimal logical edits in rulebases.
We find that the models trained in prior works do not perform consistently on the different perturbations in RobustLR.
- Score: 25.319674132967553
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Transformers have been shown to be able to perform deductive reasoning on a
logical rulebase containing rules and statements written in English natural
language. While the progress is promising, it is currently unclear if these
models indeed perform logical reasoning by understanding the underlying logical
semantics in the language. To this end, we propose RobustLR, a suite of
evaluation datasets that evaluate the robustness of these models to minimal
logical edits in rulebases and some standard logical equivalence conditions. In
our experiments with RoBERTa and T5, we find that the models trained in prior
works do not perform consistently on the different perturbations in RobustLR,
thus showing that the models are not robust to the proposed logical
perturbations. Further, we find that the models find it especially hard to
learn logical negation and disjunction operators. Overall, using our evaluation
sets, we demonstrate some shortcomings of the deductive reasoning-based
language models, which can eventually help towards designing better models for
logical reasoning over natural language.
Related papers
- Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars [0.6537995248511139]
We present a declarative framework with flexible context-sensitive rules binding multiple languages.
We construct first-order logic problems by selecting up to 32 premises and one hypothesis.
We demonstrate that using semantic constraints during generation and careful English verbalization of predicates enhances logical reasoning without hurting natural English tasks.
arXiv Detail & Related papers (2024-06-16T18:10:49Z) - LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models [52.03659714625452]
Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks.
But, can they really "reason" over the natural language?
This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied.
arXiv Detail & Related papers (2024-04-23T21:08:49Z) - NL2FOL: Translating Natural Language to First-Order Logic for Logical Fallacy Detection [45.28949266878263]
We design a process to reliably detect logical fallacies by translating natural language to First-order Logic.
We then utilize Satisfiability Modulo Theory (SMT) solvers to reason about the validity of the formula.
Our approach is robust, interpretable and does not require training data or fine-tuning.
arXiv Detail & Related papers (2024-04-18T00:20:48Z) - Language Models can be Logical Solvers [99.40649402395725]
We introduce LoGiPT, a novel language model that directly emulates the reasoning processes of logical solvers.
LoGiPT is fine-tuned on a newly constructed instruction-tuning dataset derived from revealing and refining the invisible reasoning process of deductive solvers.
arXiv Detail & Related papers (2023-11-10T16:23:50Z) - Empower Nested Boolean Logic via Self-Supervised Curriculum Learning [67.46052028752327]
We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested logic.
To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method textitCurriculum Logical Reasoning (textscClr)
arXiv Detail & Related papers (2023-10-09T06:54:02Z) - MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure [129.8481568648651]
We propose a benchmark to investigate models' logical reasoning capabilities in complex real-life scenarios.
Based on the multi-hop chain of reasoning, the explanation form includes three main components.
We evaluate the current best models' performance on this new explanation form.
arXiv Detail & Related papers (2022-10-22T16:01:13Z) - LogiGAN: Learning Logical Reasoning via Adversarial Pre-training [58.11043285534766]
We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models.
Inspired by the facilitation effect of reflective thinking in human learning, we simulate the learning-thinking process with an adversarial Generator-Verifier architecture.
Both base and large size language models pre-trained with LogiGAN demonstrate obvious performance improvement on 12 datasets.
arXiv Detail & Related papers (2022-05-18T08:46:49Z) - FaiRR: Faithful and Robust Deductive Reasoning over Natural Language [25.319674132967553]
We frame the deductive logical reasoning task by defining three modular components: rule selection, fact selection, and knowledge composition.
We observe that FaiRR is robust to novel language perturbations, and is faster at inference than previous works on existing reasoning datasets.
arXiv Detail & Related papers (2022-03-19T07:18:13Z) - Neuro-Symbolic Inductive Logic Programming with Logical Neural Networks [65.23508422635862]
We propose learning rules with the recently proposed logical neural networks (LNN)
Compared to others, LNNs offer strong connection to classical Boolean logic.
Our experiments on standard benchmarking tasks confirm that LNN rules are highly interpretable.
arXiv Detail & Related papers (2021-12-06T19:38:30Z) - Flexible Operations for Natural Language Deduction [32.92866195461153]
ParaPattern is a method for building models to generate logical transformations of diverse natural language inputs without direct human supervision.
We use a BART-based model to generate the result of applying a particular logical operation to one or more premise statements.
We evaluate our models using targeted contrast sets as well as out-of-domain sentence compositions from the QASC dataset.
arXiv Detail & Related papers (2021-04-18T11:36:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.