CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge

• URL: http://arxiv.org/abs/2403.01395v1
• Date: Sun, 3 Mar 2024 04:47:01 GMT
• Title: CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge
• Authors: Willis Guo, Armin Toroghi, Scott Sanner
• Abstract summary: We create a novel Commonsense Reasoning (CR) and Long-Tail (LT) KGQA dataset with two subtasks -- question answering and claim verification. While existing KGQA methods are not applicable due to their lack of commonsense inference support, baseline evaluation of LLMs on CR-LT KGQA demonstrate a high rate of hallucination.
• Score: 21.73770363188049
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Knowledge graph question answering (KGQA) is a well-established field that seeks to provide factual answers to natural language (NL) questions by leveraging knowledge graphs (KGs). However, existing KGQA datasets suffer from two significant limitations: (1) no existing KGQA dataset requires commonsense reasoning to arrive at an answer and (2) existing KGQA datasets focus on popular entities for which large language models (LLMs) can directly answer without hallucinating and without leveraging the KG. In this work, we seek a novel KGQA dataset that supports commonsense reasoning and focuses on long-tail entities (e.g., non-mainstream and recent entities) where LLMs frequently hallucinate, and thus create the need for novel methodologies that leverage the KG for factual and attributable commonsense inference. We create a novel Commonsense Reasoning (CR) and Long-Tail (LT) KGQA dataset with two subtasks -- question answering and claim verification -- that address both limitations (1) and (2). We construct CR-LT-KGQA by building extensions to existing reasoning datasets StrategyQA and CREAK over Wikidata. While existing KGQA methods are not applicable due to their lack of commonsense inference support, baseline evaluation of LLMs on CR-LT KGQA demonstrate a high rate of hallucination. Thus, CR-LT KGQA poses significant challenges for hallucination-prone LLMs, hence paving the way for future commonsense KGQA research to provide accurate and factual answers for long-tail entities in the era of LLMs.

Related papers

• Ontology-Guided, Hybrid Prompt Learning for Generalization in Knowledge Graph Question Answering [6.232269207752904]
  We present OntoSCPrompt, a novel Large Language Model (LLM)-based KGQA approach with a two-stage architecture. OntoSCPrompt first generates a SPARQL query structure (including SPARQL keywords such as SELECT, ASK, WHERE and placeholders for missing tokens) and then fills them with KG-specific information. We present several task-specific decoding strategies to ensure the correctness and executability of generated SPARQL queries in both stages.
  arXiv  Detail & Related papers  (2025-02-06T11:47:58Z)
• KG-CF: Knowledge Graph Completion with Context Filtering under the Guidance of Large Language Models [55.39134076436266]
  KG-CF is a framework tailored for ranking-based knowledge graph completion tasks. KG-CF leverages LLMs' reasoning abilities to filter out irrelevant contexts, achieving superior results on real-world datasets.
  arXiv  Detail & Related papers  (2025-01-06T01:52:15Z)
• Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains [66.55612528039894]
  Knowledge Graphs (KGs) can serve as reliable knowledge sources for question answering (QA) We present DoG (Decoding on Graphs), a novel framework that facilitates a deep synergy between LLMs and KGs. Experiments across various KGQA tasks with different background KGs demonstrate that DoG achieves superior and robust performance.
  arXiv  Detail & Related papers  (2024-10-24T04:01:40Z)
• LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering [1.5238808518078564]
  LinkQ is a system that leverages a large language model (LLM) to facilitate knowledge graph (KG) query construction through natural language question-answering. LLM interprets a user's question, then systematically converts it into a well-formed query. LinkQ guards against the LLM hallucinating outputs by ensuring users' questions are only ever answered from ground truth KG data.
  arXiv  Detail & Related papers  (2024-06-07T15:28:31Z)
• Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering [87.67177556994525]
  We propose a training-free method called Generate-on-Graph (GoG) to generate new factual triples while exploring Knowledge Graphs (KGs) GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA.
  arXiv  Detail & Related papers  (2024-04-23T04:47:22Z)
• Automatic Question-Answer Generation for Long-Tail Knowledge [65.11554185687258]
  We propose an automatic approach to generate specialized QA datasets for tail entities. We conduct extensive experiments by employing pretrained LLMs on our newly generated long-tail QA datasets.
  arXiv  Detail & Related papers  (2024-03-03T03:06:31Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• Improving Embedded Knowledge Graph Multi-hop Question Answering by introducing Relational Chain Reasoning [8.05076085499457]
  Knowledge Base Question Answer (KBQA) to answer userquestions from a knowledge base (KB) by identifying reasoning between topic entity and answer. As a complex branchtask of KBQA, multi-hop KGQA requires reasoning over multi-hop relational chains preserved in structured KG.
  arXiv  Detail & Related papers  (2021-10-25T06:53:02Z)
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering [122.84513233992422]
  We propose a new model, QA-GNN, which addresses the problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) We show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning.
  arXiv  Detail & Related papers  (2021-04-13T17:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.