Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

• URL: http://arxiv.org/abs/2008.09036v2
• Date: Wed, 21 Apr 2021 18:06:11 GMT
• Title: Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
• Authors: Benjamin Heinzerling and Kentaro Inui
• Abstract summary: Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. Here, we formulate two basic requirements for treating LMs as KBs. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs.
• Score: 35.57443199012129
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose single-token name is found in common LM vocabularies. Furthermore, the main benefit of this paradigm, namely querying the KB using a variety of natural language paraphrases, is underexplored so far. Here, we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.

Related papers

• Language Representations Can be What Recommenders Need: Findings and Potentials [57.90679739598295]
  We show that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation. Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.
  arXiv  Detail & Related papers  (2024-07-07T17:05:24Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Can Language Models Act as Knowledge Bases at Scale? [24.99538360485476]
  Large language models (LLMs) have demonstrated remarkable proficiency in understanding and generating responses to complex queries. Our research investigates whether LLMs can effectively store, recall, and reason with knowledge on a large scale comparable to latest knowledge bases (KBs) such as Wikidata.
  arXiv  Detail & Related papers  (2024-02-22T04:20:14Z)
• How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering [52.86931192259096]
  Knowledge Base Question Answering (KBQA) aims to answer natural language questions based on facts in knowledge bases. Recent works leverage the capabilities of large language models (LLMs) for logical form generation to improve performance.
  arXiv  Detail & Related papers  (2024-01-11T09:27:50Z)
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition [4.779196219827507]
  We argue that some of the most prominent benchmarks for evaluating the syntactic capacities of neural language models may not be sufficiently rigorous. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.
  arXiv  Detail & Related papers  (2023-10-31T00:16:17Z)
• LLM2KB: Constructing Knowledge Bases using instruction tuned context aware Large Language Models [0.8702432681310401]
  Our paper proposes LLM2KB, a system for constructing knowledge bases using large language models. Our best performing model achieved an average F1 score of 0.6185 across 21 relations in the LM-KBC challenge held at the ISWC 2023 conference.
  arXiv  Detail & Related papers  (2023-08-25T07:04:16Z)
• Language Model Analysis for Ontology Subsumption Inference [37.00562636991463]
  We investigate whether pre-trained language models (LMs) can function as knowledge bases (KBs) We propose OntoLAMA, a set of inference-based probing tasks and datasets from subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) We will open-source our code and datasets.
  arXiv  Detail & Related papers  (2023-02-14T00:21:56Z)
• A Review on Language Models as Knowledge Bases [55.035030134703995]
  Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs)
  arXiv  Detail & Related papers  (2022-04-12T18:35:23Z)
• KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models [28.82149012250609]
  We propose a benchmark, named Knowledge Memorization, Identification, and Reasoning test (KMIR) KMIR covers 3 types of knowledge, including general knowledge, domain-specific knowledge, and commonsense, and provides 184,348 well-designed questions. Preliminary experiments with various representative pre-training language models on KMIR reveal many interesting phenomenons.
  arXiv  Detail & Related papers  (2022-02-28T03:52:57Z)
• Prix-LM: Pretraining for Multilingual Knowledge Base Construction [59.02868906044296]
  We propose a unified framework, Prix-LM, for multilingual knowledge construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs. Experiments on standard entity-related tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2021-10-16T02:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.