Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs

• URL: http://arxiv.org/abs/2401.01711v1
• Date: Wed, 3 Jan 2024 12:28:33 GMT
• Title: Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs
• Authors: Phillip Schneider, Manuel Klettner, Kristiina Jokinen, Elena Simperl, Florian Matthes
• Abstract summary: This paper evaluates the performance of large language models that have not been explicitly pre-trained on this task. Our results demonstrate that large language models are capable of generating graph queries from dialogues.
• Score: 6.869834883252353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational question answering systems often rely on semantic parsing to enable interactive information retrieval, which involves the generation of structured database queries from a natural language input. For information-seeking conversations about facts stored within a knowledge graph, dialogue utterances are transformed into graph queries in a process that is called knowledge-based conversational question answering. This paper evaluates the performance of large language models that have not been explicitly pre-trained on this task. Through a series of experiments on an extensive benchmark dataset, we compare models of varying sizes with different prompting techniques and identify common issue types in the generated output. Our results demonstrate that large language models are capable of generating graph queries from dialogues, with significant improvements achievable through few-shot prompting and fine-tuning techniques, especially for smaller models that exhibit lower zero-shot performance.

Related papers

• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation [5.661396828160973]
  We conduct an empirical analysis of conversational large language models in generating natural language text from semantic triples. We compare four large language models of varying sizes with different prompting techniques. Our findings show that the capabilities of large language models in triple verbalization can be significantly improved through few-shot prompting, post-processing, and efficient fine-tuning techniques.
  arXiv  Detail & Related papers  (2024-02-02T15:26:39Z)
• OLaLa: Ontology Matching with Large Language Models [2.211868306499727]
  Ontology Matching is a challenging task where information in natural language is one of the most important signals to process. With the rise of Large Language Models, it is possible to incorporate this knowledge in a better way into the matching pipeline. We show that with only a handful of examples and a well-designed prompt, it is possible to achieve results that are en par with supervised matching systems.
  arXiv  Detail & Related papers  (2023-11-07T09:34:20Z)
• Conversational Semantic Parsing using Dynamic Context Graphs [68.72121830563906]
  We consider the task of conversational semantic parsing over general purpose knowledge graphs (KGs) with millions of entities, and thousands of relation-types. We focus on models which are capable of interactively mapping user utterances into executable logical forms.
  arXiv  Detail & Related papers  (2023-05-04T16:04:41Z)
• Semantic Parsing for Conversational Question Answering over Knowledge Graphs [63.939700311269156]
  We develop a dataset where user questions are annotated with Sparql parses and system answers correspond to execution results thereof. We present two different semantic parsing approaches and highlight the challenges of the task. Our dataset and models are released at https://github.com/Edinburgh/SPICE.
  arXiv  Detail & Related papers  (2023-01-28T14:45:11Z)
• Reasoning in Dialog: Improving Response Generation by Context Reading Comprehension [49.92173751203827]
  In multi-turn dialog, utterances do not always take the full form of sentences. We propose to improve the response generation performance by examining the model's ability to answer a reading comprehension question.
  arXiv  Detail & Related papers  (2020-12-14T10:58:01Z)
• Plug-and-Play Conversational Models [62.77150879036442]
  We introduce an approach that does not require further computation at decoding time, while also does not require any fine-tuning of a large language model. We demonstrate, through extensive automatic and human evaluation, a high degree of control over the generated conversational responses with regard to multiple desired attributes.
  arXiv  Detail & Related papers  (2020-10-09T03:17:51Z)
• MLR: A Two-stage Conversational Query Rewriting Model with Multi-task Learning [16.88648782206587]
  We propose the conversational query rewriting model - MLR, which is a Multi-task model on sequence Labeling and query Rewriting. MLR reformulates the multi-turn conversational queries into a single turn query, which conveys the true intention of users concisely. To train our model, we construct a new Chinese query rewriting dataset and conduct experiments on it.
  arXiv  Detail & Related papers  (2020-04-13T08:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.