Logical Reasoning for Natural Language Inference Using Generated Facts
as Atoms
- URL: http://arxiv.org/abs/2305.13214v1
- Date: Mon, 22 May 2023 16:45:50 GMT
- Title: Logical Reasoning for Natural Language Inference Using Generated Facts
as Atoms
- Authors: Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu
and Marek Rei
- Abstract summary: We introduce a model-agnostic logical framework to determine the specific information in an input responsible for each model decision.
This method creates interpretable Natural Language Inference (NLI) models that maintain their predictive power.
- Score: 26.286055953538284
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: State-of-the-art neural models can now reach human performance levels across
various natural language understanding tasks. However, despite this impressive
performance, models are known to learn from annotation artefacts at the expense
of the underlying task. While interpretability methods can identify influential
features for each prediction, there are no guarantees that these features are
responsible for the model decisions. Instead, we introduce a model-agnostic
logical framework to determine the specific information in an input responsible
for each model decision. This method creates interpretable Natural Language
Inference (NLI) models that maintain their predictive power. We achieve this by
generating facts that decompose complex NLI observations into individual
logical atoms. Our model makes predictions for each atom and uses logical rules
to decide the class of the observation based on the predictions for each atom.
We apply our method to the highly challenging ANLI dataset, where our framework
improves the performance of both a DeBERTa-base and BERT baseline. Our method
performs best on the most challenging examples, achieving a new
state-of-the-art for the ANLI round 3 test set. We outperform every baseline in
a reduced-data setting, and despite using no annotations for the generated
facts, our model predictions for individual facts align with human
expectations.
Related papers
- Preserving Knowledge Invariance: Rethinking Robustness Evaluation of
Open Information Extraction [50.62245481416744]
We present the first benchmark that simulates the evaluation of open information extraction models in the real world.
We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.
By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
arXiv Detail & Related papers (2023-05-23T12:05:09Z) - Multi-Scales Data Augmentation Approach In Natural Language Inference
For Artifacts Mitigation And Pre-Trained Model Optimization [0.0]
We provide a variety of techniques for analyzing and locating dataset artifacts inside the crowdsourced Stanford Natural Language Inference corpus.
To mitigate dataset artifacts, we employ a unique multi-scale data augmentation technique with two distinct frameworks.
Our combination method enhances our model's resistance to perturbation testing, enabling it to continuously outperform the pre-trained baseline.
arXiv Detail & Related papers (2022-12-16T23:37:44Z) - Large Language Models with Controllable Working Memory [64.71038763708161]
Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP)
What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining.
How the model's world knowledge interacts with the factual information presented in the context remains under explored.
arXiv Detail & Related papers (2022-11-09T18:58:29Z) - Logical Reasoning with Span Predictions: Span-level Logical Atoms for
Interpretable and Robust NLI Models [19.601700560645206]
Current Natural Language Inference (NLI) models achieve impressive results, sometimes outperforming humans on in-distribution test sets.
We introduce a logical reasoning framework for NLI, creating highly transparent model decisions that are based on logical rules.
We almost fully retain performance on SNLI while identifying the exact hypothesis spans that are responsible for each model prediction.
arXiv Detail & Related papers (2022-05-23T16:24:27Z) - An Empirical Investigation of Commonsense Self-Supervision with
Knowledge Graphs [67.23285413610243]
Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models.
We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
arXiv Detail & Related papers (2022-05-21T19:49:04Z) - Explaining the Deep Natural Language Processing by Mining Textual
Interpretable Features [3.819533618886143]
T-EBAnO is a prediction-local and class-based model-global explanation strategies tailored to deep natural-language models.
It provides an objective, human-readable, domain-specific assessment of the reasons behind the automatic decision-making process.
arXiv Detail & Related papers (2021-06-12T06:25:09Z) - Unnatural Language Inference [48.45003475966808]
We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words.
Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
arXiv Detail & Related papers (2020-12-30T20:40:48Z) - PSD2 Explainable AI Model for Credit Scoring [0.0]
The aim of this project is to develop and test advanced analytical methods to improve the prediction accuracy of Credit Risk Models.
The project focuses on applying an explainable machine learning model to bank-related databases.
arXiv Detail & Related papers (2020-11-20T12:12:38Z) - Explaining and Improving Model Behavior with k Nearest Neighbor
Representations [107.24850861390196]
We propose using k nearest neighbor representations to identify training examples responsible for a model's predictions.
We show that kNN representations are effective at uncovering learned spurious associations.
Our results indicate that the kNN approach makes the finetuned model more robust to adversarial inputs.
arXiv Detail & Related papers (2020-10-18T16:55:25Z) - Rethinking Generalization of Neural Models: A Named Entity Recognition
Case Study [81.11161697133095]
We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives.
Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models.
As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
arXiv Detail & Related papers (2020-01-12T04:33:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.