Chain of Logic: Rule-Based Reasoning with Large Language Models

• URL: http://arxiv.org/abs/2402.10400v2
• Date: Fri, 23 Feb 2024 18:55:23 GMT
• Title: Chain of Logic: Rule-Based Reasoning with Large Language Models
• Authors: Sergio Servantez, Joe Barrow, Kristian Hammond, Rajiv Jain
• Abstract summary: Rule-based reasoning enables us to draw conclusions by accurately applying a rule to a set of facts. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition and recomposition. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark.
• Score: 10.017812995997753
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rule-based reasoning, a fundamental type of legal reasoning, enables us to draw conclusions by accurately applying a rule to a set of facts. We explore causal language models as rule-based reasoners, specifically with respect to compositional rules - rules consisting of multiple elements which form a complex logical expression. Reasoning about compositional rules is challenging because it requires multiple reasoning steps, and attending to the logical relationships between elements. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to resolve the underlying logical expression). This method was inspired by the IRAC (Issue, Rule, Application, Conclusion) framework, a sequential reasoning approach used by lawyers. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark and demonstrate it consistently outperforms other prompting methods, including chain of thought and self-ask, using open-source and commercial language models.

Related papers

• Explaining Non-monotonic Normative Reasoning using Argumentation Theory with Deontic Logic [7.162465547358201]
  This paper explores how to provide designers with effective explanations for their legally relevant design decisions. We extend the previous system for providing explanations by specifying norms and the key legal or ethical principles for justifying actions in normative contexts. Considering that first-order logic has strong expressive power, in the current paper we adopt a first-order deontic logic system with deontic operators and preferences.
  arXiv  Detail & Related papers  (2024-09-18T08:03:29Z)
• LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models [87.49676980090555]
  Large Language Models (LLMs) have demonstrated notable capabilities across various tasks, showcasing complex problem-solving abilities. We introduce LogicGame, a novel benchmark designed to evaluate the comprehensive rule understanding, execution, and planning capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-08-28T13:16:41Z)
• LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models [52.03659714625452]
  Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks. But, can they really "reason" over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied.
  arXiv  Detail & Related papers  (2024-04-23T21:08:49Z)
• Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs [87.34281749422756]
  Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. We propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic.
  arXiv  Detail & Related papers  (2024-02-18T03:38:51Z)
• ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning [107.61997887260056]
  We propose a novel framework, ChatRule, unleashing the power of large language models for mining logical rules over knowledge graphs. Specifically, the framework is initiated with an LLM-based rule generator, leveraging both the semantic and structural information of KGs. To refine the generated rules, a rule ranking module estimates the rule quality by incorporating facts from existing KGs.
  arXiv  Detail & Related papers  (2023-09-04T11:38:02Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)
• Argumentative Characterizations of (Extended) Disjunctive Logic Programs [2.055949720959582]
  We show that assumption-based argumentation can represent not only normal logic programs, but also disjunctive logic programs and their extensions. We consider some inference rules for disjunction that the core logic of the argumentation frameworks should respect.
  arXiv  Detail & Related papers  (2023-06-12T14:01:38Z)
• Deontic Meta-Rules [2.241042010144441]
  This work extends such a logical framework by considering the deontic aspect. The resulting logic will not just be able to model policies but also tackle well-known aspects that occur in numerous legal systems.
  arXiv  Detail & Related papers  (2022-09-23T07:48:29Z)
• Automating Defeasible Reasoning in Law [0.0]
  We study defeasible reasoning in rule-based systems, in particular about legal norms and contracts. We identify rule modifier that specify how rules interact and how they can be overridden. We then define rule transformations that eliminate these modifier, leading to a translation of rules to formulas.
  arXiv  Detail & Related papers  (2022-05-15T17:14:15Z)
• RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs [91.71504177786792]
  This paper studies learning logic rules for reasoning on knowledge graphs. Logic rules provide interpretable explanations when used for prediction as well as being able to generalize to other tasks. Existing methods either suffer from the problem of searching in a large search space or ineffective optimization due to sparse rewards.
  arXiv  Detail & Related papers  (2020-10-08T14:47:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.