CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models

• URL: http://arxiv.org/abs/2009.13964v5
• Date: Wed, 5 Apr 2023 07:55:56 GMT
• Title: CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models
• Authors: Yusheng Su, Xu Han, Zhengyan Zhang, Peng Li, Zhiyuan Liu, Yankai Lin, Jie Zhou and Maosong Sun
• Abstract summary: We propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context. Our experimental results show that Coke outperforms various baselines on typical knowledge-driven NLP tasks. Coke can describe the semantics of text-related knowledge in a more interpretable form than the conventional PLMs.
• Score: 103.18329049830152
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Several recent efforts have been devoted to enhancing pre-trained language models (PLMs) by utilizing extra heterogeneous knowledge in knowledge graphs (KGs) and achieved consistent improvements on various knowledge-driven NLP tasks. However, most of these knowledge-enhanced PLMs embed static sub-graphs of KGs ("knowledge context"), regardless of that the knowledge required by PLMs may change dynamically according to specific text ("textual context"). In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text. Our experimental results show that Coke outperforms various baselines on typical knowledge-driven NLP tasks, indicating the effectiveness of utilizing dynamic knowledge context for language understanding. Besides the performance improvements, the dynamically selected knowledge in Coke can describe the semantics of text-related knowledge in a more interpretable form than the conventional PLMs. Our source code and datasets will be available to provide more details for Coke.

Related papers

• Knowledge Graph-Enhanced Large Language Models via Path Selection [58.228392005755026]
  Large Language Models (LLMs) have shown unprecedented performance in various real-world applications. LLMs are known to generate factually inaccurate outputs, a.k.a. the hallucination problem. We propose a principled framework KELP with three stages to handle the above problems.
  arXiv  Detail & Related papers  (2024-06-19T21:45:20Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• A Survey of Knowledge Enhanced Pre-trained Language Models [78.56931125512295]
  We present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG) and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods.
  arXiv  Detail & Related papers  (2022-11-11T04:29:02Z)
• DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning [18.838291575019504]
  Pre-trained language models (PLMs) are shown to be lacking in knowledge when dealing with knowledge driven tasks. We propose textbfDictBERT, a novel approach that enhances PLMs with dictionary knowledge. We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE.
  arXiv  Detail & Related papers  (2022-08-01T06:43:19Z)
• CoSe-Co: Text Conditioned Generative CommonSense Contextualizer [13.451001884972033]
  Pre-trained Language Models (PTLMs) have been shown to perform well on natural language tasks. We propose a CommonSense Contextualizer (CoSe-Co) conditioned on sentences as input to make it generically usable in tasks.
  arXiv  Detail & Related papers  (2022-06-12T09:57:32Z)
• ECOLA: Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations [35.51427298619691]
  We study enhancing temporal knowledge embedding with textual data. We propose Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA) Experiments show that ECOLA significantly enhances temporal embedding models with up to 287% relative improvements regarding Hits@1 on the link prediction task.
  arXiv  Detail & Related papers  (2022-03-17T20:08:25Z)
• TegTok: Augmenting Text Generation via Task-specific and Open-world Knowledge [83.55215993730326]
  We propose augmenting TExt Generation via Task-specific and Open-world Knowledge (TegTok) in a unified framework. Our model selects knowledge entries from two types of knowledge sources through dense retrieval and then injects them into the input encoding and output decoding stages respectively.
  arXiv  Detail & Related papers  (2022-03-16T10:37:59Z)
• KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs [26.557447199727758]
  We propose a novel knowledge-aware language model framework based on fine-tuning process. Our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT.
  arXiv  Detail & Related papers  (2021-09-09T12:39:17Z)
• CoLAKE: Contextualized Language and Knowledge Embedding [81.90416952762803]
  We propose the Contextualized Language and Knowledge Embedding (CoLAKE) CoLAKE jointly learns contextualized representation for both language and knowledge with the extended objective. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks.
  arXiv  Detail & Related papers  (2020-10-01T11:39:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.