Decomposing Natural Logic Inferences in Neural NLI

• URL: http://arxiv.org/abs/2112.08289v2
• Date: Wed, 8 Nov 2023 13:31:39 GMT
• Title: Decomposing Natural Logic Inferences in Neural NLI
• Authors: Julia Rozanova, Deborah Ferreira, Marco Valentino, Mokanrarangan Thayaparan, Andre Freitas
• Abstract summary: We investigate whether neural NLI models capture the crucial semantic features central to natural logic: monotonicity and concept inclusion. We find that monotonicity information is notably weak in the representations of popular NLI models which achieve high scores on benchmarks.
• Score: 9.606462437067984
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the interest of interpreting neural NLI models and their reasoning strategies, we carry out a systematic probing study which investigates whether these models capture the crucial semantic features central to natural logic: monotonicity and concept inclusion. Correctly identifying valid inferences in downward-monotone contexts is a known stumbling block for NLI performance, subsuming linguistic phenomena such as negation scope and generalized quantifiers. To understand this difficulty, we emphasize monotonicity as a property of a context and examine the extent to which models capture monotonicity information in the contextual embeddings which are intermediate to their decision making process. Drawing on the recent advancement of the probing paradigm, we compare the presence of monotonicity features across various models. We find that monotonicity information is notably weak in the representations of popular NLI models which achieve high scores on benchmarks, and observe that previous improvements to these models based on fine-tuning strategies have introduced stronger monotonicity features together with their improved performance on challenge sets.

Related papers

• Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness [68.69369585600698]
  Deep learning models often suffer from a lack of interpretability due to polysemanticity. Recent advances in monosemanticity, where neurons correspond to consistent and distinct semantics, have significantly improved interpretability. We show that monosemantic features not only enhance interpretability but also bring concrete gains in model performance.
  arXiv  Detail & Related papers  (2024-10-27T18:03:20Z)
• Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
  In this study, we explore employing neural networks as ensemble methods. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• MonoKAN: Certified Monotonic Kolmogorov-Arnold Network [48.623199394622546]
  In certain applications, model predictions must align with expert-imposed requirements, sometimes exemplified by partial monotonicity constraints. We introduce a novel ANN architecture called MonoKAN, based on the KAN architecture and achieves certified partial monotonicity while enhancing interpretability. Our experiments demonstrate that MonoKAN not only enhances interpretability but also improves predictive performance across the majority of benchmarks, outperforming state-of-the-art monotonic approaches.
  arXiv  Detail & Related papers  (2024-09-17T11:10:59Z)
• Enhancing adversarial robustness in Natural Language Inference using explanations [41.46494686136601]
  We cast the spotlight on the underexplored task of Natural Language Inference (NLI) We validate the usage of natural language explanation as a model-agnostic defence strategy through extensive experimentation. We research the correlation of widely used language generation metrics with human perception, in order for them to serve as a proxy towards robust NLI models.
  arXiv  Detail & Related papers  (2024-09-11T17:09:49Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective [72.55093886515824]
  We introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables. We devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a causal graph. Experiment results on synthetic and real datasets show that our three proposed components significantly improve the robustness and reusability of the learned motion representations.
  arXiv  Detail & Related papers  (2021-11-29T18:59:09Z)
• Supporting Context Monotonicity Abstractions in Neural NLI Models [2.624902795082451]
  In certain NLI problems, the entailment label depends only on the context monotonicity and the relation between the substituted concepts. We introduce a sound and complete simplified monotonicity logic formalism which describes our treatment of contexts as abstract units. Using the notions in our formalism, we adapt targeted challenge sets to investigate whether an intermediate context monotonicity classification task can aid NLI models' performance.
  arXiv  Detail & Related papers  (2021-05-17T16:43:43Z)
• Exploring Transitivity in Neural NLI Models through Veridicality [39.845425535943534]
  We focus on the transitivity of inference relations, a fundamental property for systematically drawing inferences. A model capturing transitivity can compose basic inference patterns and draw new inferences. We find that current NLI models do not perform consistently well on transitivity inference tasks.
  arXiv  Detail & Related papers  (2021-01-26T11:18:35Z)
• Exploring End-to-End Differentiable Natural Logic Modeling [21.994060519995855]
  We explore end-to-end trained differentiable models that integrate natural logic with neural networks. The proposed model adapts module networks to model natural logic operations, which is enhanced with a memory component to model contextual information.
  arXiv  Detail & Related papers  (2020-11-08T18:18:15Z)
• Explaining and Improving Model Behavior with k Nearest Neighbor Representations [107.24850861390196]
  We propose using k nearest neighbor representations to identify training examples responsible for a model's predictions. We show that kNN representations are effective at uncovering learned spurious associations. Our results indicate that the kNN approach makes the finetuned model more robust to adversarial inputs.
  arXiv  Detail & Related papers  (2020-10-18T16:55:25Z)
• Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation [14.431925736607043]
  We present Monotonicity NLI (MoNLI), a new naturalistic dataset focused on lexical entailment and negation. In behavioral evaluations, we find that models trained on general-purpose NLI datasets fail systematically on MoNLI examples containing negation. In structural evaluations, we look for evidence that our top-performing BERT-based model has learned to implement the monotonicity algorithm behind MoNLI.
  arXiv  Detail & Related papers  (2020-04-30T07:53:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.