Evaluating Language Models for Knowledge Base Completion

• URL: http://arxiv.org/abs/2303.11082v1
• Date: Mon, 20 Mar 2023 13:14:59 GMT
• Title: Evaluating Language Models for Knowledge Base Completion
• Abstract summary: Language models (LMs) have recently been proposed for unsupervised knowledge base completion (KBC) We introduce a novel, more challenging benchmark dataset, and a methodology tailored for a realistic assessment of the KBC potential of LMs. We find that LMs possess surprisingly strong generalization capabilities, even on relations where most facts were not directly observed in LM training.
• Score: 32.87131159997359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Structured knowledge bases (KBs) are a foundation of many intelligent applications, yet are notoriously incomplete. Language models (LMs) have recently been proposed for unsupervised knowledge base completion (KBC), yet, despite encouraging initial results, questions regarding their suitability remain open. Existing evaluations often fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we introduce a novel, more challenging benchmark dataset, and a methodology tailored for a realistic assessment of the KBC potential of LMs. For automated assessment, we curate a dataset called WD-KNOWN, which provides an unbiased random sample of Wikidata, containing over 3.9 million facts. In a second step, we perform a human evaluation on predictions that are not yet in the KB, as only this provides real insights into the added value over existing KBs. Our key finding is that biases in dataset conception of previous benchmarks lead to a systematic overestimate of LM performance for KBC. However, our results also reveal strong areas of LMs. We could, for example, perform a significant completion of Wikidata on the relations nativeLanguage, by a factor of ~21 (from 260k to 5.8M) at 82% precision, usedLanguage, by a factor of ~2.1 (from 2.1M to 6.6M) at 82% precision, and citizenOf by a factor of ~0.3 (from 4.2M to 5.3M) at 90% precision. Moreover, we find that LMs possess surprisingly strong generalization capabilities: even on relations where most facts were not directly observed in LM training, prediction quality can be high.

Related papers

• GPTKB 2.0: Direct Construction of Disambiguated Knowledge Bases from Large Language Models [11.119171720690238]
  GPTKB 2.0 is a methodology for constructing disambiguated KBs directly from large language models.<n>We execute GPTKB 2.0 at scale, obtaining a materialized KB containing over 1M disambiguated entities and 38.4M triples.
  arXiv  Detail & Related papers  (2026-08-04T14:25:47Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data [42.23843583401247]
  Real-world settings require language models to grapple with incomplete information and reason under uncertainty.<n>OpenEstimate is a benchmark for evaluating LMs on numerical estimation.<n>We find that LM-elicited priors are often inaccurate and overconfident.
  arXiv  Detail & Related papers  (2025-10-16T19:35:22Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages? [35.368257850926184]
  We focus on a zero-shot evaluation setting focusing on low-resource Indian languages, namely Assamese, Kannada, Maithili, and Punjabi. We observe that even for learned metrics, which are known to exhibit zero-shot performance, the Kendall Tau and Pearson correlations with human annotations are only as high as 0.32 and 0.45.
  arXiv  Detail & Related papers  (2024-06-06T09:28:08Z)
• Evaluating the Knowledge Base Completion Potential of GPT [29.75708908603862]
  GPT-3 enables to extend Wikidata by 27M facts at 90% precision. We show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.
  arXiv  Detail & Related papers  (2023-10-23T10:15:13Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)
• A Review on Language Models as Knowledge Bases [55.035030134703995]
  Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs)
  arXiv  Detail & Related papers  (2022-04-12T18:35:23Z)
• Ranking vs. Classifying: Measuring Knowledge Base Completion Quality [10.06803520598035]
  We argue that consideration of binary predictions is essential to reflect the actual KBC quality. We simulate the realistic scenario of real-world entities missing from a KB. We evaluate a number of state-of-the-art KB embeddings models on our new benchmark.
  arXiv  Detail & Related papers  (2021-02-02T17:53:48Z)
• Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases [63.43418760818188]
  We release a new large-scale, high-quality dataset with 64,331 questions, GrailQA. We propose a novel BERT-based KBQA model. The combination of our dataset and model enables us to thoroughly examine and demonstrate, for the first time, the key role of pre-trained contextual embeddings like BERT in the generalization of KBQA.
  arXiv  Detail & Related papers  (2020-11-16T06:36:26Z)
• Improving Candidate Generation for Low-resource Cross-lingual Entity Linking [81.41804263432684]
  Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. In this paper, we propose three improvements that (1) reduce the disconnect between entity mentions and KB entries, and (2) improve the robustness of the model to low-resource scenarios.
  arXiv  Detail & Related papers  (2020-03-03T05:32:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.