Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!

• URL: http://arxiv.org/abs/2405.11706v1
• Date: Mon, 20 May 2024 00:28:00 GMT
• Title: Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!
• Authors: Dean Allemang, Juan Sequeda,
• Abstract summary: We present an approach that consists of 1) Ontology-based Query Check (OBQC) and 2) LLM Repair. Our approach increases the overall accuracy to 72% including an additional 8% of "I don't know" unknown results.
• Score: 1.0786522863027366
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There is increasing evidence that question-answering (QA) systems with Large Language Models (LLMs), which employ a knowledge graph/semantic representation of an enterprise SQL database (i.e. Text-to-SPARQL), achieve higher accuracy compared to systems that answer questions directly on SQL databases (i.e. Text-to-SQL). Our previous benchmark research showed that by using a knowledge graph, the accuracy improved from 16% to 54%. The question remains: how can we further improve the accuracy and reduce the error rate? Building on the observations of our previous research where the inaccurate LLM-generated SPARQL queries followed incorrect paths, we present an approach that consists of 1) Ontology-based Query Check (OBQC): detects errors by leveraging the ontology of the knowledge graph to check if the LLM-generated SPARQL query matches the semantic of ontology and 2) LLM Repair: use the error explanations with an LLM to repair the SPARQL query. Using the chat with the data benchmark, our primary finding is that our approach increases the overall accuracy to 72% including an additional 8% of "I don't know" unknown results. Thus, the overall error rate is 20%. These results provide further evidence that investing knowledge graphs, namely the ontology, provides higher accuracy for LLM powered question answering systems.

Related papers

• Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling [69.84963245729826]
  Large language models (LLMs) have shown compelling semantic understanding capabilities. Dense retrieval is a crucial task in Information Retrieval (IR) and is the foundation for downstream tasks as re-ranking. We introduce an auxiliary task of QL estimation to yield a better backbone for contrast learning a discriminative retriever.
  arXiv  Detail & Related papers  (2025-04-07T16:03:59Z)
• QE-RAG: A Robust Retrieval-Augmented Generation Benchmark for Query Entry Errors [23.225358970952197]
  Retriever-augmented generation (RAG) has become a widely adopted approach for enhancing the factual accuracy of large language models (LLMs) QE-RAG is the first robust RAG benchmark designed specifically to evaluate performance against query entry errors. We propose a contrastive learning-based robust retriever training method and a retrieval-augmented query correction method.
  arXiv  Detail & Related papers  (2025-04-05T05:24:08Z)
• Reducing Hallucinations in Language Model-based SPARQL Query Generation Using Post-Generation Memory Retrieval [9.860751439256754]
  Large language models (LLMs) are susceptible to hallucinations and out-of-distribution errors when producing KG elements. This has led to increased research aimed at detecting and mitigating such errors. In this paper, we introduce PGMR, a modular framework that incorporates a non-parametric memory module to retrieve KG elements.
  arXiv  Detail & Related papers  (2025-02-19T02:08:13Z)
• Are LLMs Really Not Knowledgable? Mining the Submerged Knowledge in LLMs' Memory [15.986679553468989]
  Large language models (LLMs) have shown promise as potential knowledge bases. LLMs often struggle with question-answering tasks and are prone to hallucinations. We develop SkipUnsure, a method to improve answer accuracy by leveraging detected but unexpressed knowledge.
  arXiv  Detail & Related papers  (2024-12-30T10:29:18Z)
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation [81.18701211912779]
  We introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. Our method has achieved state-of-the-art performance on two common datasets.
  arXiv  Detail & Related papers  (2024-12-24T16:38:04Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• Traceable LLM-based validation of statements in knowledge graphs [0.0]
  This article presents a method for verifying RDF triples using LLMs, with an emphasis on providing traceable arguments. Instead, verified RDF statements are compared to chunks of external documents retrieved through a web search or Wikipedia. To assess the possible application of this workflow on biosciences content, we evaluated 1,719 positive statements from the BioRED dataset and the same number of newly generated negative statements.
  arXiv  Detail & Related papers  (2024-09-11T12:27:41Z)
• S-EQA: Tackling Situational Queries in Embodied Question Answering [48.43453390717167]
  We present and tackle the problem of Embodied Question Answering with Situational Queries (S-EQA) in a household environment. We first introduce a novel Prompt-Generate-Evaluate scheme that wraps around an LLM's output to create a dataset of unique situational queries and corresponding consensus object information. We report an improved accuracy of 15.31% while using queries framed from the generated object consensus for Visual Question Answering (VQA) over directly answering situational ones.
  arXiv  Detail & Related papers  (2024-05-08T00:45:20Z)
• Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs [102.37496443389203]
  'Logic-Query-of-Thoughts' (LGOT) is the first of its kind to combine knowledge graph reasoning and large language models. Our experimental findings demonstrate substantial performance enhancements, with up to 20% improvement over ChatGPT.
  arXiv  Detail & Related papers  (2024-03-17T17:01:45Z)
• Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis [4.412170175171256]
  This study investigates various approaches to using Large Language Models for Text-to- program synthesis. The goal was to input a natural language question along with the database schema and output the correct SELECT query.
  arXiv  Detail & Related papers  (2024-01-22T22:05:42Z)
• Leveraging LLMs in Scholarly Knowledge Graph Question Answering [7.951847862547378]
  KGQA answers natural language questions by leveraging a large language model (LLM) Our system achieves an F1 score of 99.0% on SciQA - one of the Scholarly Knowledge Graph Question Answering challenge benchmarks.
  arXiv  Detail & Related papers  (2023-11-16T12:13:49Z)
• A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases [1.0786522863027366]
  This study aims to evaluate the accuracy of LLM-powered question answering systems in the context of enterprise questions. We also explore the role of knowledge graphs in improving accuracy.
  arXiv  Detail & Related papers  (2023-11-13T17:54:50Z)
• Learning From Mistakes Makes LLM Better Reasoner [106.48571828587728]
  Large language models (LLMs) recently exhibited remarkable reasoning capabilities on solving math problems. This work explores whether LLMs can LEarn from MistAkes (LEMA), akin to the human learning process.
  arXiv  Detail & Related papers  (2023-10-31T17:52:22Z)
• Can Large Language Models Infer Causation from Correlation? [104.96351414570239]
  We test the pure causal inference skills of large language models (LLMs) We formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We show that these models achieve almost close to random performance on the task.
  arXiv  Detail & Related papers  (2023-06-09T12:09:15Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.