Enhancing Systematic Decompositional Natural Language Inference Using
Informal Logic
- URL: http://arxiv.org/abs/2402.14798v2
- Date: Tue, 27 Feb 2024 21:53:24 GMT
- Title: Enhancing Systematic Decompositional Natural Language Inference Using
Informal Logic
- Authors: Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei
Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen,
Peter Clark, Benjamin Van Durme
- Abstract summary: We develop a consistent and theoretically grounded approach to annotating decompositional entailment datasets.
We find that our resulting dataset, RDTE, has a substantially higher internal consistency (+9%) than prior decompositional entailment datasets.
We also find that training an RDTE-oriented entailment classifier via knowledge distillation and employing it in a modern neuro-symbolic reasoning engine significantly improves results.
- Score: 53.363888563647976
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Contemporary language models enable new opportunities for structured
reasoning with text, such as the construction and evaluation of intuitive,
proof-like textual entailment trees without relying on brittle formal logic.
However, progress in this direction has been hampered by a long-standing lack
of a clear protocol for determining what valid compositional entailment is.
This absence causes noisy datasets and limited performance gains by modern
neuro-symbolic engines. To address these problems, we formulate a consistent
and theoretically grounded approach to annotating decompositional entailment
datasets, and evaluate its impact on LLM-based textual inference. We find that
our resulting dataset, RDTE (Recognizing Decompositional Textual Entailment),
has a substantially higher internal consistency (+9%) than prior
decompositional entailment datasets, suggesting that RDTE is a significant step
forward in the long-standing problem of forming a clear protocol for discerning
entailment. We also find that training an RDTE-oriented entailment classifier
via knowledge distillation and employing it in a modern neuro-symbolic
reasoning engine significantly improves results (both accuracy and proof
quality) over other entailment classifier baselines, illustrating the practical
benefit of this advance for textual inference.
Related papers
- Deja vu: Contrastive Historical Modeling with Prefix-tuning for Temporal Knowledge Graph Reasoning [16.408149489677154]
ChapTER is a Contrastive historical modeling framework with prefix-tuning for TEmporal Reasoning.
We evaluate ChapTER on four transductive and three few-shot inductive TKGR benchmarks.
arXiv Detail & Related papers (2024-03-25T17:25:40Z) - Multi-modal Causal Structure Learning and Root Cause Analysis [67.67578590390907]
We propose Mulan, a unified multi-modal causal structure learning method for root cause localization.
We leverage a log-tailored language model to facilitate log representation learning, converting log sequences into time-series data.
We also introduce a novel key performance indicator-aware attention mechanism for assessing modality reliability and co-learning a final causal graph.
arXiv Detail & Related papers (2024-02-04T05:50:38Z) - How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored.
Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges.
We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
arXiv Detail & Related papers (2023-11-14T08:51:00Z) - Prompt-based Logical Semantics Enhancement for Implicit Discourse
Relation Recognition [4.7938839332508945]
We propose a Prompt-based Logical Semantics Enhancement (PLSE) method for Implicit Discourse Relation Recognition (IDRR)
Our method seamlessly injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction.
Experimental results on PDTB 2.0 and CoNLL16 datasets demonstrate that our method achieves outstanding and consistent performance against the current state-of-the-art models.
arXiv Detail & Related papers (2023-11-01T08:38:08Z) - Learning a Structural Causal Model for Intuition Reasoning in
Conversation [20.243323155177766]
Reasoning, a crucial aspect of NLP research, has not been adequately addressed by prevailing models.
We develop a conversation cognitive model ( CCM) that explains how each utterance receives and activates channels of information.
By leveraging variational inference, it explores substitutes for implicit causes, addresses the issue of their unobservability, and reconstructs the causal representations of utterances through the evidence lower bounds.
arXiv Detail & Related papers (2023-05-28T13:54:09Z) - CausalDialogue: Modeling Utterance-level Causality in Conversations [83.03604651485327]
We have compiled and expanded upon a new dataset called CausalDialogue through crowd-sourcing.
This dataset includes multiple cause-effect pairs within a directed acyclic graph (DAG) structure.
We propose a causality-enhanced method called Exponential Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models.
arXiv Detail & Related papers (2022-12-20T18:31:50Z) - MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning [63.50909998372667]
We propose MERIt, a MEta-path guided contrastive learning method for logical ReasonIng of text.
Two novel strategies serve as indispensable components of our method.
arXiv Detail & Related papers (2022-03-01T11:13:00Z) - A Dependency Syntactic Knowledge Augmented Interactive Architecture for
End-to-End Aspect-based Sentiment Analysis [73.74885246830611]
We propose a novel dependency syntactic knowledge augmented interactive architecture with multi-task learning for end-to-end ABSA.
This model is capable of fully exploiting the syntactic knowledge (dependency relations and types) by leveraging a well-designed Dependency Relation Embedded Graph Convolutional Network (DreGcn)
Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2020-04-04T14:59:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.