Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata

• URL: http://arxiv.org/abs/2305.14202v2
• Date: Sun, 5 Nov 2023 19:26:17 GMT
• Title: Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata
• Authors: Silei Xu, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina J. Semnani, Monica S. Lam
• Abstract summary: This paper presents WikiWebQuestions, a high-quality question answering benchmark for Wikidata. It consists of real-world data with SPARQL. We modify SPARQL to use the unique domain and property names instead of their IDs.
• Score: 6.716263690738313
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While large language models (LLMs) can answer many questions correctly, they can also hallucinate and give wrong answers. Wikidata, with its over 12 billion facts, can be used to ground LLMs to improve their factuality. This paper presents WikiWebQuestions, a high-quality question answering benchmark for Wikidata. Ported over from WebQuestions for Freebase, it consists of real-world data with SPARQL annotation. This paper presents a few-shot sequence-to-sequence semantic parser for Wikidata. We modify SPARQL to use the unique domain and property names instead of their IDs. We train the parser to use either the results from an entity linker or mentions in the query. We fine-tune LLaMA by adding the few-shot training data to that used to fine-tune Alpaca. Our experimental results demonstrate the effectiveness of this methodology, establishing a strong baseline of 76% and 65% answer accuracy in the dev and test sets of WikiWebQuestions, respectively. By pairing our semantic parser with GPT-3, we combine verifiable results with qualified GPT-3 guesses to provide useful answers to 96% of the questions in dev. We also show that our method outperforms the state-of-the-art for the QALD-7 Wikidata dataset by 3.6% in F1 score.

Related papers

• KIF: A Wikidata-Based Framework for Integrating Heterogeneous Knowledge Sources [0.45141207783683707]
  We present a Wikidata-based framework, called KIF, for virtually integrating heterogeneous knowledge sources. KIF is written in Python and is released as open-source.
  arXiv  Detail & Related papers  (2024-03-15T13:46:36Z)
• KGConv, a Conversational Corpus grounded in Wikidata [6.451914896767135]
  KGConv is a large, conversational corpus of 71k conversations grounded in a Wikidata fact. We provide multiple variants (12 on average) of the corresponding question using templates, human annotations, hand-crafted rules and a question rewriting neural model. KGConv can further be used for other generation and analysis tasks such as single-turn question generation from Wikidata triples, question rewriting, question answering from conversation or from knowledge graphs and quiz generation.
  arXiv  Detail & Related papers  (2023-08-29T13:35:51Z)
• Semantic Parsing for Conversational Question Answering over Knowledge Graphs [63.939700311269156]
  We develop a dataset where user questions are annotated with Sparql parses and system answers correspond to execution results thereof. We present two different semantic parsing approaches and highlight the challenges of the task. Our dataset and models are released at https://github.com/Edinburgh/SPICE.
  arXiv  Detail & Related papers  (2023-01-28T14:45:11Z)
• Mapping Process for the Task: Wikidata Statements to Text as Wikipedia Sentences [68.8204255655161]
  We propose our mapping process for the task of converting Wikidata statements to natural language text (WS2T) for Wikipedia projects at the sentence level. The main step is to organize statements, represented as a group of quadruples and triples, and then to map them to corresponding sentences in English Wikipedia. We evaluate the output corpus in various aspects: sentence structure analysis, noise filtering, and relationships between sentence components based on word embedding models.
  arXiv  Detail & Related papers  (2022-10-23T08:34:33Z)
• WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs [66.88232442007062]
  We introduce WikiDes, a dataset to generate short descriptions of Wikipedia articles. The dataset consists of over 80k English samples on 6987 topics. Our paper shows a practical impact on Wikipedia and Wikidata since there are thousands of missing descriptions.
  arXiv  Detail & Related papers  (2022-09-27T01:28:02Z)
• Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking [76.00737707718795]
  We propose a novel candidate retrieval paradigm based on entity profiling. We use the profile to query the indexed search engine to retrieve candidate entities. Our approach complements the traditional approach of using a Wikipedia anchor-text dictionary.
  arXiv  Detail & Related papers  (2022-02-27T17:38:53Z)
• QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers [68.9964449363406]
  We extend one of the most popular KGQA benchmarks - QALD-9 by introducing high-quality questions' translations to 8 languages. Five of the languages - Armenian, Ukrainian, Lithuanian, Bashkir and Belarusian - to our best knowledge were never considered in KGQA research community before.
  arXiv  Detail & Related papers  (2022-01-31T22:19:55Z)
• Assessing the quality of sources in Wikidata across languages: a hybrid approach [64.05097584373979]
  We run a series of microtasks experiments to evaluate a large corpus of references, sampled from Wikidata triples with labels in several languages. We use a consolidated, curated version of the crowdsourced assessments to train several machine learning models to scale up the analysis to the whole of Wikidata. The findings help us ascertain the quality of references in Wikidata, and identify common challenges in defining and capturing the quality of user-generated multilingual structured data on the web.
  arXiv  Detail & Related papers  (2021-09-20T10:06:46Z)
• Creating and Querying Personalized Versions of Wikidata on a Laptop [0.7449724123186383]
  This paper introduces KGTK Kypher, a query language and processor that allows users to create personalized variants of Wikidata on a laptop. We present several use cases that illustrate the types of analyses that Kypher enables users to run on the full Wikidata KG on a laptop.
  arXiv  Detail & Related papers  (2021-08-06T00:00:33Z)
• Commonsense Knowledge in Wikidata [3.8359194344969807]
  This paper investigates whether Wikidata con-tains commonsense knowledge which is complementary to existing commonsense sources. We map the relations of Wikidata to ConceptNet, which we also leverage to integrate Wikidata-CS into an existing consolidated commonsense graph.
  arXiv  Detail & Related papers  (2020-08-18T18:23:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.