Knowledge Authoring for Rules and Actions

• URL: http://arxiv.org/abs/2305.07763v1
• Date: Fri, 12 May 2023 21:08:35 GMT
• Title: Knowledge Authoring for Rules and Actions
• Authors: Yuheng Wang, Paul Fodor, Michael Kifer
• Abstract summary: We propose KALMRA to enable authoring of rules and actions. Our evaluation shows that KALMRA achieves a high level of correctness (100%) on rule authoring. We illustrate the logical reasoning capabilities of KALMRA by drawing attention to the problems faced by the recently made famous AI, ChatGPT.
• Score: 1.942275677807562
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge representation and reasoning (KRR) systems describe and reason with complex concepts and relations in the form of facts and rules. Unfortunately, wide deployment of KRR systems runs into the problem that domain experts have great difficulty constructing correct logical representations of their domain knowledge. Knowledge engineers can help with this construction process, but there is a deficit of such specialists. The earlier Knowledge Authoring Logic Machine (KALM) based on Controlled Natural Language (CNL) was shown to have very high accuracy for authoring facts and questions. More recently, KALMFL, a successor of KALM, replaced CNL with factual English, which is much less restrictive and requires very little training from users. However, KALMFL has limitations in representing certain types of knowledge, such as authoring rules for multi-step reasoning or understanding actions with timestamps. To address these limitations, we propose KALMRA to enable authoring of rules and actions. Our evaluation using the UTI guidelines benchmark shows that KALMRA achieves a high level of correctness (100%) on rule authoring. When used for authoring and reasoning with actions, KALMRA achieves more than 99.3% correctness on the bAbI benchmark, demonstrating its effectiveness in more sophisticated KRR jobs. Finally, we illustrate the logical reasoning capabilities of KALMRA by drawing attention to the problems faced by the recently made famous AI, ChatGPT.

Related papers

• Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning [34.427730009102966]
  We develop an automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs. Our work will also serve as an evaluation framework that can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks.
  arXiv  Detail & Related papers  (2025-02-08T19:49:32Z)
• RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios [58.90106984375913]
  RuleArena is a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions.
  arXiv  Detail & Related papers  (2024-12-12T06:08:46Z)
• Knowledge Authoring with Factual English, Rules, and Actions [1.2110885481490308]
  CNL-based approaches have shown to have very high accuracy compared to others. We propose KALM for Rules and Actions (KALMR) to represent and reason with rules and actions. When used for authoring and reasoning with actions, our approach achieves more than 99.3% correctness.
  arXiv  Detail & Related papers  (2024-11-09T19:01:34Z)
• Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning [87.10396098919013]
  Large Language Models (LLMs) have demonstrated extensive knowledge and remarkable proficiency in temporal reasoning. We propose a Large Language Models-guided Dynamic Adaptation (LLM-DA) method for reasoning on Temporal Knowledge Graphs. LLM-DA harnesses the capabilities of LLMs to analyze historical data and extract temporal logical rules.
  arXiv  Detail & Related papers  (2024-05-23T04:54:37Z)
• Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs [87.34281749422756]
  Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. We propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic.
  arXiv  Detail & Related papers  (2024-02-18T03:38:51Z)
• DeepEdit: Knowledge Editing as Decoding with Constraints [118.78008395850888]
  How to edit the knowledge in multi-step reasoning has become the major challenge in the knowledge editing (KE) of large language models (LLMs) We propose a new KE framework: DEEPEDIT, which enhances LLMs's ability to generate coherent reasoning chains with new knowledge through depth-first search. In addition to DEEPEDIT, we propose two new KE benchmarks: MQUAKE-2002 and MQUAKE-HARD, which provide more precise and challenging assessments of KE approaches.
  arXiv  Detail & Related papers  (2024-01-19T03:48:27Z)
• Temporal Knowledge Question Answering via Abstract Reasoning Induction [32.08799860090592]
  This study addresses the challenge of enhancing temporal knowledge reasoning in Large Language Models (LLMs) We propose Abstract Reasoning Induction (ARI) framework, which divides temporal reasoning into two distinct phases: Knowledge-agnostic and Knowledge-based. Our approach achieves remarkable improvements, with relative gains of 29.7% and 9.27% on two temporal QA datasets.
  arXiv  Detail & Related papers  (2023-11-15T17:46:39Z)
• Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning [104.92384929827776]
  Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. They lack up-to-date knowledge and experience hallucinations during reasoning. Knowledge graphs (KGs) offer a reliable source of knowledge for reasoning.
  arXiv  Detail & Related papers  (2023-10-02T10:14:43Z)
• Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models [56.34029644009297]
  Large language models (LLMs) have demonstrated the ability to overcome various limitations of formal Knowledge Representation (KR) systems. LLMs excel most in abductive reasoning, followed by deductive reasoning, while they are least effective at inductive reasoning. We study single-task training, multi-task training, and "chain-of-thought" knowledge distillation fine-tuning technique to assess the performance of model.
  arXiv  Detail & Related papers  (2023-10-02T01:00:50Z)
• ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning [107.61997887260056]
  We propose a novel framework, ChatRule, unleashing the power of large language models for mining logical rules over knowledge graphs. Specifically, the framework is initiated with an LLM-based rule generator, leveraging both the semantic and structural information of KGs. To refine the generated rules, a rule ranking module estimates the rule quality by incorporating facts from existing KGs.
  arXiv  Detail & Related papers  (2023-09-04T11:38:02Z)
• Knowledge Authoring with Factual English [0.0]
  Knowledge representation and reasoning (KRR) systems represent knowledge as collections of facts and rules. One solution could be to extract knowledge from English text, and a number of works have attempted to do so. Unfortunately, extraction of logical facts from unrestricted natural language is still too inaccurate to be used for reasoning. Recent CNL-based approaches, such as the Knowledge Authoring Logic Machine (KALM), have shown to have very high accuracy compared to others.
  arXiv  Detail & Related papers  (2022-08-05T10:49:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.