Related papers: A Review on Language Models as Knowledge Bases

A Review on Language Models as Knowledge Bases

URL: http://arxiv.org/abs/2204.06031v1
Date: Tue, 12 Apr 2022 18:35:23 GMT
Title: A Review on Language Models as Knowledge Bases
Authors: Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, Marjan Ghazvininejad
Abstract summary: Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs)
Score: 55.035030134703995
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem a LM should have to fully act as a KB, and review the recent literature with respect to those aspects.

Related papers

A Knowledge-Injected Curriculum Pretraining Framework for Question Answering [70.13026036388794]
We propose a general Knowledge-Injected Curriculum Pretraining framework (KICP) to achieve comprehensive KG learning and exploitation for Knowledge-based question answering tasks. The KI module first injects knowledge into the LM by generating KG-centered pretraining corpus, and generalizes the process into three key steps. The KA module learns knowledge from the generated corpus with LM equipped with an adapter as well as keeps its original natural language understanding ability. The CR module follows human reasoning patterns to construct three corpora with increasing difficulties of reasoning, and further trains the LM from easy to hard in a curriculum manner.
arXiv Detail & Related papers (2024-03-11T03:42:03Z)
Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models [57.60487455727155]
idioms, with their non-compositional nature, pose particular challenges for Transformer-based systems. Traditional methods, which replace idioms using existing knowledge bases (KBs), often lack scale and context awareness. We introduce a multilingual idiom KB (IdiomKB) developed using large LMs to address this. This KB facilitates better translation by smaller models, such as BLOOMZ (7.1B), Alpaca (7B), and InstructGPT (6.7B)
arXiv Detail & Related papers (2023-08-26T21:38:31Z)
KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases [55.942342665806656]
KnowledGPT is a comprehensive framework to bridge large language models with various knowledge bases. The retrieval process employs the program of thought prompting, which generates search language for KBs in code format. KnowledGPT offers the capability to store knowledge in a personalized KB, catering to individual user demands.
arXiv Detail & Related papers (2023-08-17T13:07:00Z)
A Survey of Knowledge Enhanced Pre-trained Language Models [78.56931125512295]
We present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG) and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods.
arXiv Detail & Related papers (2022-11-11T04:29:02Z)
Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Base [51.55027623439027]
We develop two solutions to help the model learn more knowledge from unstructured text in a fully self-supervised manner. To our best knowledge, we are the first to explore fully self-supervised learning of knowledge in continual pre-training.
arXiv Detail & Related papers (2022-04-17T12:33:34Z)
Language Models As or For Knowledge Bases [30.089955948497405]
We identify strengths and limitations of pre-trained language models (LMs) and explicit knowledge bases (KBs) We argue that latent LMs are not suitable as a substitute for explicit KBs, but could play a major role for augmenting and curating KBs.
arXiv Detail & Related papers (2021-10-10T20:00:09Z)
Relational world knowledge representation in contextual language models: A review [19.176173014629185]
We take a natural language processing perspective to the limitations of knowledge bases (KBs) We propose a novel taxonomy for relational knowledge representation in contextual language models (LMs)
arXiv Detail & Related papers (2021-04-12T21:50:55Z)
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries [35.57443199012129]
Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. Here, we formulate two basic requirements for treating LMs as KBs. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs.
arXiv Detail & Related papers (2020-08-20T15:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.