Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering

• URL: http://arxiv.org/abs/2403.01390v2
• Date: Fri, 11 Oct 2024 02:55:15 GMT
• Title: Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering
• Authors: Armin Toroghi, Willis Guo, Mohammad Mahdi Abdollah Pour, Scott Sanner,
• Abstract summary: Knowledge Graph Question Answering (KGQA) methods seek to answer Natural Language questions using the relational information stored in Knowledge Graphs (KGs) With the recent advancements of Large Language Models (LLMs) and their remarkable reasoning abilities, there is a growing trend to leverage them for KGQA. We propose Right for Right Reasons (R3), a commonsense KGQA methodology that allows for a verifiable reasoning procedure.
• Score: 18.48602809114524
• License:
• Abstract: Knowledge Graph Question Answering (KGQA) methods seek to answer Natural Language questions using the relational information stored in Knowledge Graphs (KGs). With the recent advancements of Large Language Models (LLMs) and their remarkable reasoning abilities, there is a growing trend to leverage them for KGQA. However, existing methodologies have only focused on answering factual questions, e.g., "In which city was Silvio Berlusconi's first wife born?", leaving questions involving commonsense reasoning that real-world users may pose more often, e.g., "Do I need separate visas to see the Venus of Willendorf and attend the Olympics this summer?" unaddressed. In this work, we first observe that existing LLM-based methods for KGQA struggle with hallucination on such questions, especially on queries targeting long-tail entities (e.g., non-mainstream and recent entities), thus hindering their applicability in real-world applications especially since their reasoning processes are not easily verifiable. In response, we propose Right for Right Reasons (R3), a commonsense KGQA methodology that allows for a verifiable reasoning procedure by axiomatically surfacing intrinsic commonsense knowledge of LLMs and grounding every factual reasoning step on KG triples. Through experimental evaluations across three different tasks--question answering, claim verification, and preference matching--our findings showcase R3 as a superior approach, outperforming existing methodologies and notably reducing instances of hallucination and reasoning errors.

Related papers

• Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models [33.662269036173456]
  Large Language Models (LLMs) may suffer from hallucinations in real-world applications due to the lack of relevant knowledge. Knowledge Graph Question Answering (KGQA) serves as a critical touchstone for the integration. We propose an interactive KGQA framework that leverages the interactive learning capabilities of LLMs to perform reasoning and Debating over Graphs (DoG)
  arXiv  Detail & Related papers  (2024-09-05T01:11:58Z)
• Evidence-Enhanced Triplet Generation Framework for Hallucination Alleviation in Generative Question Answering [41.990482015732574]
  We propose a novel evidence-enhanced triplet generation framework, EATQA, to predict all the combinations of (Question, Evidence, Answer) triplet. We bridge the distribution gap to distill the knowledge from evidence in inference stage. Our framework ensures the model to learn the logical relation between query, evidence and answer, which simultaneously improves the evidence generation and query answering.
  arXiv  Detail & Related papers  (2024-08-27T13:07:07Z)
• Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering [87.67177556994525]
  We propose a training-free method called Generate-on-Graph (GoG) to generate new factual triples while exploring Knowledge Graphs (KGs) GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA.
  arXiv  Detail & Related papers  (2024-04-23T04:47:22Z)
• Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs [102.37496443389203]
  'Logic-Query-of-Thoughts' (LGOT) is the first of its kind to combine knowledge graph reasoning and large language models. Our experimental findings demonstrate substantial performance enhancements, with up to 20% improvement over ChatGPT.
  arXiv  Detail & Related papers  (2024-03-17T17:01:45Z)
• Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning [104.92384929827776]
  Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. They lack up-to-date knowledge and experience hallucinations during reasoning. Knowledge graphs (KGs) offer a reliable source of knowledge for reasoning.
  arXiv  Detail & Related papers  (2023-10-02T10:14:43Z)
• Graph Reasoning for Question Answering with Triplet Retrieval [33.454090126152714]
  We propose a simple yet effective method to retrieve the most relevant triplets from knowledge graphs (KGs) Our method can outperform state-of-the-art up to 4.6% absolute accuracy.
  arXiv  Detail & Related papers  (2023-05-30T04:46:28Z)
• Multi-hop Commonsense Knowledge Injection Framework for Zero-Shot Commonsense Question Answering [6.086719709100659]
  We propose a novel multi-hop commonsense knowledge injection framework. Our framework achieves state-of-art performance on five commonsense question answering benchmarks.
  arXiv  Detail & Related papers  (2023-05-10T07:13:47Z)
• Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks [121.74957524305283]
  This paper proposes a novel framework named textbfSearch-in-the-Chain (SearChain) for the interaction between Information Retrieval (IR) and Large Language Model (LLM) Experiments show that SearChain outperforms state-of-the-art baselines on complex knowledge-intensive tasks.
  arXiv  Detail & Related papers  (2023-04-28T10:15:25Z)
• WikiWhy: Answering and Explaining Cause-and-Effect Questions [62.60993594814305]
  We introduce WikiWhy, a QA dataset built around explaining why an answer is true in natural language. WikiWhy contains over 9,000 "why" question-answer-rationale triples, grounded on Wikipedia facts across a diverse set of topics. GPT-3 baselines achieve only 38.7% human-evaluated correctness in the end-to-end answer & explain condition.
  arXiv  Detail & Related papers  (2022-10-21T17:59:03Z)
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering [159.9645181522436]
  GreaseLM is a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. We show that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
  arXiv  Detail & Related papers  (2022-01-21T19:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.