LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models

• URL: http://arxiv.org/abs/2203.15099v1
• Date: Mon, 28 Mar 2022 21:13:22 GMT
• Title: LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models
• Authors: Santiago Ontanon, Joshua Ainslie, Vaclav Cvicek and Zachary Fisher
• Abstract summary: This paper presents LogicInference, a new dataset to evaluate the ability of models to perform logical inference. The dataset focuses on inference using propositional logic and a small subset of first-order logic. We also report initial results using a collection of machine learning models to establish an initial baseline in this dataset.
• Score: 4.186923466475792
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning models such as Transformers or LSTMs struggle with tasks that are compositional in nature such as those involving reasoning/inference. Although many datasets exist to evaluate compositional generalization, when it comes to evaluating inference abilities, options are more limited. This paper presents LogicInference, a new dataset to evaluate the ability of models to perform logical inference. The dataset focuses on inference using propositional logic and a small subset of first-order logic, represented both in semi-formal logical notation, as well as in natural language. We also report initial results using a collection of machine learning models to establish an initial baseline in this dataset.

Related papers

• Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning [25.496627355906966]
  We develop three new logical reasoning datasets named "ReClor-plus", "LogiQA-plus" and "LogiQAv2-plus" Experiments show that these simple augmentations greatly hinder the models' performance. Applying logic-driven data augmentation for fine-tuning and prompting can enhance generalisation in both discriminative and generative models.
  arXiv  Detail & Related papers  (2023-10-13T22:29:15Z)
• LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning [73.98142349171552]
  LOGICSEG is a holistic visual semantic that integrates neural inductive learning and logic reasoning with both rich data and symbolic knowledge. During fuzzy logic-based continuous relaxation, logical formulae are grounded onto data and neural computational graphs, hence enabling logic-induced network training. These designs together make LOGICSEG a general and compact neural-logic machine that is readily integrated into existing segmentation models.
  arXiv  Detail & Related papers  (2023-09-24T05:43:19Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)
• APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning [73.3035118224719]
  We propose APOLLO, an adaptively pretrained language model that has improved logical reasoning abilities. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA.
  arXiv  Detail & Related papers  (2022-12-19T07:40:02Z)
• Investigating the Robustness of Natural Language Generation from Logical Forms via Counterfactual Samples [30.079030298066847]
  State-of-the-art methods based on pre-trained models have achieved remarkable performance on the standard test dataset. We question whether these methods really learn how to perform logical reasoning, rather than just relying on the spurious correlations between the headers of the tables and operators of the logical form. We propose two approaches to reduce the model's reliance on the shortcut.
  arXiv  Detail & Related papers  (2022-10-16T14:14:53Z)
• PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation [44.78200830757109]
  We propose a PLOG (Pretrained Logical Form Generator) framework to improve the generation fidelity. PLOG is first pretrained on a table-to-logic-form generation task, then finetuned on downstream table-to-text tasks. PLOG can learn logical inference from table-logic pairs much more definitely than from table-text pairs.
  arXiv  Detail & Related papers  (2022-05-25T11:55:54Z)
• Structural Learning of Probabilistic Sentential Decision Diagrams under Partial Closed-World Assumption [127.439030701253]
  Probabilistic sentential decision diagrams are a class of structured-decomposable circuits. We propose a new scheme based on a partial closed-world assumption: data implicitly provide the logical base of the circuit. Preliminary experiments show that the proposed approach might properly fit training data, and generalize well to test data, provided that these remain consistent with the underlying logical base.
  arXiv  Detail & Related papers  (2021-07-26T12:01:56Z)
• A Topological-Framework to Improve Analysis of Machine Learning Model Performance [5.3893373617126565]
  We propose a framework for evaluating machine learning models in which a dataset is treated as a "space" on which a model operates. We describe a topological data structure, presheaves, which offer a convenient way to store and analyze model performance between different subpopulations.
  arXiv  Detail & Related papers  (2021-07-09T23:11:13Z)
• Logical Natural Language Generation from Open-Domain Tables [107.04385677577862]
  We propose a new task where a model is tasked with generating natural language statements that can be emphlogically entailed by the facts. To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset citechen 2019tabfact featured with a wide range of logical/symbolic inferences. The new task poses challenges to the existing monotonic generation frameworks due to the mismatch between sequence order and logical order.
  arXiv  Detail & Related papers  (2020-04-22T06:03:10Z)
• Evaluating Logical Generalization in Graph Neural Networks [59.70452462833374]
  We study the task of logical generalization using graph neural networks (GNNs) Our benchmark suite, GraphLog, requires that learning algorithms perform rule induction in different synthetic logics. We find that the ability for models to generalize and adapt is strongly determined by the diversity of the logical rules they encounter during training.
  arXiv  Detail & Related papers  (2020-03-14T05:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.