Evaluating the Knowledge Base Completion Potential of GPT

• URL: http://arxiv.org/abs/2310.14771v1
• Date: Mon, 23 Oct 2023 10:15:13 GMT
• Title: Evaluating the Knowledge Base Completion Potential of GPT
• Abstract summary: GPT-3 enables to extend Wikidata by 27M facts at 90% precision. We show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.
• Score: 29.75708908603862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we perform a careful evaluation of GPT's potential to complete the largest public KB: Wikidata. We find that, despite their size and capabilities, models like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, they provide solid improvements over earlier approaches with smaller LMs. In particular, we show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.

Related papers

• GPTKB: Building Very Large Knowledge Bases from Language Models [15.276041443299905]
  We propose to build a large general-domain KB entirely from a large language model (LLM) As a prototype, we use GPT-4o-mini to construct GPTKB, which contains 105 million triples for more than 2.9 million entities. For NLP, for the first time, it provides textitconstructive insights into the knowledge (or beliefs) of LLMs.
  arXiv  Detail & Related papers  (2024-11-07T17:57:03Z)
• Large Language Models as Reliable Knowledge Bases? [60.25969380388974]
  Large Language Models (LLMs) can be viewed as potential knowledge bases (KBs) This study defines criteria that a reliable LLM-as-KB should meet, focusing on factuality and consistency. strategies like ICL and fine-tuning are unsuccessful at making LLMs better KBs.
  arXiv  Detail & Related papers  (2024-07-18T15:20:18Z)
• Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models [59.52207546810294]
  Listwise rerankers based on large language models (LLM) are the zero-shot state-of-art. In this work, we build for the first time effective listwise rerankers without any form of dependency on GPT. Our best list se reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves 97% effectiveness of the ones built on GPT-4.
  arXiv  Detail & Related papers  (2023-12-05T18:57:40Z)
• Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts [21.150221839202878]
  Large Language Models (LLMs) have achieved significant success across various general tasks. In this work, we examine the proficiency of LLMs in generating succinct survey articles specific to the niche field of NLP in computer science. We compare both human and GPT-based evaluation scores and provide in-depth analysis.
  arXiv  Detail & Related papers  (2023-08-21T01:32:45Z)
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models [92.6951708781736]
  This work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5. We find that GPT models can be easily misled to generate toxic and biased outputs and leak private information. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps.
  arXiv  Detail & Related papers  (2023-06-20T17:24:23Z)
• Evaluating Language Models for Knowledge Base Completion [32.87131159997359]
  Language models (LMs) have recently been proposed for unsupervised knowledge base completion (KBC) We introduce a novel, more challenging benchmark dataset, and a methodology tailored for a realistic assessment of the KBC potential of LMs. We find that LMs possess surprisingly strong generalization capabilities, even on relations where most facts were not directly observed in LM training.
  arXiv  Detail & Related papers  (2023-03-20T13:14:59Z)
• Prompting GPT-3 To Be Reliable [117.23966502293796]
  This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality. We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
  arXiv  Detail & Related papers  (2022-10-17T14:52:39Z)
• Prompting as Probing: Using Language Models for Knowledge Base Construction [1.6050172226234583]
  We present ProP (Prompting as Probing), which utilizes GPT-3, a large Language Model originally proposed by OpenAI in 2020. ProP implements a multi-step approach that combines a variety of prompting techniques to achieve this. Our evaluation study indicates that these proposed techniques can substantially enhance the quality of the final predictions.
  arXiv  Detail & Related papers  (2022-08-23T16:03:50Z)
• A Review on Language Models as Knowledge Bases [55.035030134703995]
  Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs)
  arXiv  Detail & Related papers  (2022-04-12T18:35:23Z)
• Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases [63.43418760818188]
  We release a new large-scale, high-quality dataset with 64,331 questions, GrailQA. We propose a novel BERT-based KBQA model. The combination of our dataset and model enables us to thoroughly examine and demonstrate, for the first time, the key role of pre-trained contextual embeddings like BERT in the generalization of KBQA.
  arXiv  Detail & Related papers  (2020-11-16T06:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.