Knowledge Prompting in Pre-trained Language Model for Natural Language
Understanding
- URL: http://arxiv.org/abs/2210.08536v1
- Date: Sun, 16 Oct 2022 13:36:57 GMT
- Title: Knowledge Prompting in Pre-trained Language Model for Natural Language
Understanding
- Authors: Jianing Wang, Wenkang Huang, Qiuhui Shi, Hongbin Wang, Minghui Qiu,
Xiang Li, Ming Gao
- Abstract summary: We propose a knowledge-prompting-based PLM framework KP-PLM.
This framework can be flexibly combined with existing mainstream PLMs.
To further leverage the factual knowledge from these prompts, we propose two novel knowledge-aware self-supervised tasks.
- Score: 24.315130086787374
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Knowledge-enhanced Pre-trained Language Model (PLM) has recently received
significant attention, which aims to incorporate factual knowledge into PLMs.
However, most existing methods modify the internal structures of fixed types of
PLMs by stacking complicated modules, and introduce redundant and irrelevant
factual knowledge from knowledge bases (KBs). In this paper, to address these
problems, we introduce a seminal knowledge prompting paradigm and further
propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be
flexibly combined with existing mainstream PLMs. Specifically, we first
construct a knowledge sub-graph from KBs for each context. Then we design
multiple continuous prompts rules and transform the knowledge sub-graph into
natural language prompts. To further leverage the factual knowledge from these
prompts, we propose two novel knowledge-aware self-supervised tasks including
prompt relevance inspection and masked prompt modeling. Extensive experiments
on multiple natural language understanding (NLU) tasks show the superiority of
KP-PLM over other state-of-the-art methods in both full-resource and
low-resource settings.
Related papers
- A Knowledge-Injected Curriculum Pretraining Framework for Question Answering [70.13026036388794]
We propose a general Knowledge-Injected Curriculum Pretraining framework (KICP) to achieve comprehensive KG learning and exploitation for Knowledge-based question answering tasks.
The KI module first injects knowledge into the LM by generating KG-centered pretraining corpus, and generalizes the process into three key steps.
The KA module learns knowledge from the generated corpus with LM equipped with an adapter as well as keeps its original natural language understanding ability.
The CR module follows human reasoning patterns to construct three corpora with increasing difficulties of reasoning, and further trains the LM from easy to hard in a curriculum manner.
arXiv Detail & Related papers (2024-03-11T03:42:03Z) - KnowledGPT: Enhancing Large Language Models with Retrieval and Storage
Access on Knowledge Bases [55.942342665806656]
KnowledGPT is a comprehensive framework to bridge large language models with various knowledge bases.
The retrieval process employs the program of thought prompting, which generates search language for KBs in code format.
KnowledGPT offers the capability to store knowledge in a personalized KB, catering to individual user demands.
arXiv Detail & Related papers (2023-08-17T13:07:00Z) - Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus.
We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
arXiv Detail & Related papers (2023-05-15T15:47:09Z) - UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language
Models [100.4659557650775]
We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge.
With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
arXiv Detail & Related papers (2023-05-02T17:33:28Z) - A Survey of Knowledge Enhanced Pre-trained Language Models [78.56931125512295]
We present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs)
For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG) and rule knowledge.
The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods.
arXiv Detail & Related papers (2022-11-11T04:29:02Z) - Knowledge Prompts: Injecting World Knowledge into Language Models
through Soft Prompts [8.425194277824996]
We introduce a method to train soft prompts via self-supervised learning on data from knowledge bases.
The resulting soft knowledge prompts (KPs) are task independent and work as an external memory of the LMs.
arXiv Detail & Related papers (2022-10-10T14:31:16Z) - DictBERT: Dictionary Description Knowledge Enhanced Language Model
Pre-training via Contrastive Learning [18.838291575019504]
Pre-trained language models (PLMs) are shown to be lacking in knowledge when dealing with knowledge driven tasks.
We propose textbfDictBERT, a novel approach that enhances PLMs with dictionary knowledge.
We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE.
arXiv Detail & Related papers (2022-08-01T06:43:19Z) - Knowledgeable Salient Span Mask for Enhancing Language Models as
Knowledge Base [51.55027623439027]
We develop two solutions to help the model learn more knowledge from unstructured text in a fully self-supervised manner.
To our best knowledge, we are the first to explore fully self-supervised learning of knowledge in continual pre-training.
arXiv Detail & Related papers (2022-04-17T12:33:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.