Decomposing Natural Logic Inferences in Neural NLI
- URL: http://arxiv.org/abs/2112.08289v2
- Date: Wed, 8 Nov 2023 13:31:39 GMT
- Title: Decomposing Natural Logic Inferences in Neural NLI
- Authors: Julia Rozanova, Deborah Ferreira, Marco Valentino, Mokanrarangan
Thayaparan, Andre Freitas
- Abstract summary: We investigate whether neural NLI models capture the crucial semantic features central to natural logic: monotonicity and concept inclusion.
We find that monotonicity information is notably weak in the representations of popular NLI models which achieve high scores on benchmarks.
- Score: 9.606462437067984
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In the interest of interpreting neural NLI models and their reasoning
strategies, we carry out a systematic probing study which investigates whether
these models capture the crucial semantic features central to natural logic:
monotonicity and concept inclusion. Correctly identifying valid inferences in
downward-monotone contexts is a known stumbling block for NLI performance,
subsuming linguistic phenomena such as negation scope and generalized
quantifiers. To understand this difficulty, we emphasize monotonicity as a
property of a context and examine the extent to which models capture
monotonicity information in the contextual embeddings which are intermediate to
their decision making process. Drawing on the recent advancement of the probing
paradigm, we compare the presence of monotonicity features across various
models. We find that monotonicity information is notably weak in the
representations of popular NLI models which achieve high scores on benchmarks,
and observe that previous improvements to these models based on fine-tuning
strategies have introduced stronger monotonicity features together with their
improved performance on challenge sets.
Related papers
- Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models [16.328341121232484]
We apply causal effect estimation strategies to measure the effect of context interventions.
We investigate robustness to irrelevant changes and sensitivity to impactful changes of Transformers.
arXiv Detail & Related papers (2024-04-03T10:22:35Z) - Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
We evaluate language understanding capacities on simple inference tasks that most humans find trivial.
We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments.
The models exhibit moderate to low performance on these evaluation sets.
arXiv Detail & Related papers (2023-05-24T06:41:09Z) - Towards Robust and Adaptive Motion Forecasting: A Causal Representation
Perspective [72.55093886515824]
We introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables.
We devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a causal graph.
Experiment results on synthetic and real datasets show that our three proposed components significantly improve the robustness and reusability of the learned motion representations.
arXiv Detail & Related papers (2021-11-29T18:59:09Z) - Supporting Context Monotonicity Abstractions in Neural NLI Models [2.624902795082451]
In certain NLI problems, the entailment label depends only on the context monotonicity and the relation between the substituted concepts.
We introduce a sound and complete simplified monotonicity logic formalism which describes our treatment of contexts as abstract units.
Using the notions in our formalism, we adapt targeted challenge sets to investigate whether an intermediate context monotonicity classification task can aid NLI models' performance.
arXiv Detail & Related papers (2021-05-17T16:43:43Z) - Exploring Transitivity in Neural NLI Models through Veridicality [39.845425535943534]
We focus on the transitivity of inference relations, a fundamental property for systematically drawing inferences.
A model capturing transitivity can compose basic inference patterns and draw new inferences.
We find that current NLI models do not perform consistently well on transitivity inference tasks.
arXiv Detail & Related papers (2021-01-26T11:18:35Z) - Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis [76.39883780990489]
We analyze the behavior of non-autoregressive TTS models under different prosody-modeling settings.
We propose a hierarchical architecture, in which the prediction of phoneme-level prosody features are conditioned on the word-level prosody features.
arXiv Detail & Related papers (2020-11-12T16:16:41Z) - Exploring End-to-End Differentiable Natural Logic Modeling [21.994060519995855]
We explore end-to-end trained differentiable models that integrate natural logic with neural networks.
The proposed model adapts module networks to model natural logic operations, which is enhanced with a memory component to model contextual information.
arXiv Detail & Related papers (2020-11-08T18:18:15Z) - Explaining and Improving Model Behavior with k Nearest Neighbor
Representations [107.24850861390196]
We propose using k nearest neighbor representations to identify training examples responsible for a model's predictions.
We show that kNN representations are effective at uncovering learned spurious associations.
Our results indicate that the kNN approach makes the finetuned model more robust to adversarial inputs.
arXiv Detail & Related papers (2020-10-18T16:55:25Z) - Understanding Neural Abstractive Summarization Models via Uncertainty [54.37665950633147]
seq2seq abstractive summarization models generate text in a free-form manner.
We study the entropy, or uncertainty, of the model's token-level predictions.
We show that uncertainty is a useful perspective for analyzing summarization and text generation models more broadly.
arXiv Detail & Related papers (2020-10-15T16:57:27Z) - High-order Semantic Role Labeling [86.29371274587146]
This paper introduces a high-order graph structure for the neural semantic role labeling model.
It enables the model to explicitly consider not only the isolated predicate-argument pairs but also the interaction between the predicate-argument pairs.
Experimental results on 7 languages of the CoNLL-2009 benchmark show that the high-order structural learning techniques are beneficial to the strong performing SRL models.
arXiv Detail & Related papers (2020-10-09T15:33:54Z) - Neural Natural Language Inference Models Partially Embed Theories of
Lexical Entailment and Negation [14.431925736607043]
We present Monotonicity NLI (MoNLI), a new naturalistic dataset focused on lexical entailment and negation.
In behavioral evaluations, we find that models trained on general-purpose NLI datasets fail systematically on MoNLI examples containing negation.
In structural evaluations, we look for evidence that our top-performing BERT-based model has learned to implement the monotonicity algorithm behind MoNLI.
arXiv Detail & Related papers (2020-04-30T07:53:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.