Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

• URL: http://arxiv.org/abs/2004.14224v1
• Date: Wed, 29 Apr 2020 14:22:42 GMT
• Title: Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
• Authors: Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, Weizhu Chen
• Abstract summary: We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
• Score: 73.0598186896953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.

Related papers

• Knowledge Graphs and Pre-trained Language Models enhanced Representation Learning for Conversational Recommender Systems [58.561904356651276]
  We introduce the Knowledge-Enhanced Entity Representation Learning (KERL) framework to improve the semantic understanding of entities for Conversational recommender systems. KERL uses a knowledge graph and a pre-trained language model to improve the semantic understanding of entities. KERL achieves state-of-the-art results in both recommendation and response generation tasks.
  arXiv  Detail & Related papers  (2023-12-18T06:41:23Z)
• CADGE: Context-Aware Dialogue Generation Enhanced with Graph-Structured Knowledge Aggregation [25.56539617837482]
  A novel context-aware graph-attention model (Context-aware GAT) is proposed. It assimilates global features from relevant knowledge graphs through a context-enhanced knowledge aggregation mechanism. Empirical results demonstrate that our framework outperforms conventional GNN-based language models in terms of performance.
  arXiv  Detail & Related papers  (2023-05-10T16:31:35Z)
• KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction [0.0]
  We introduce a new entity/relation embedding layer that learns to differentiate distinctive entity and relation types. We show that further pre-training the language models with this additional embedding layer using the triples extracted from the knowledge graph, followed by the standard fine-tuning phase sets a new state-of-the-art performance for the link prediction task on the benchmark datasets.
  arXiv  Detail & Related papers  (2022-11-04T20:38:12Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• Joint Language Semantic and Structure Embedding for Knowledge Graph Completion [66.15933600765835]
  We propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models. Our experiments on a variety of knowledge graph benchmarks have demonstrated the state-of-the-art performance of our method.
  arXiv  Detail & Related papers  (2022-09-19T02:41:02Z)
• KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs [26.557447199727758]
  We propose a novel knowledge-aware language model framework based on fine-tuning process. Our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT.
  arXiv  Detail & Related papers  (2021-09-09T12:39:17Z)
• JAKET: Joint Pre-training of Knowledge Graph and Language Understanding [73.43768772121985]
  We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains.
  arXiv  Detail & Related papers  (2020-10-02T05:53:36Z)
• CoLAKE: Contextualized Language and Knowledge Embedding [81.90416952762803]
  We propose the Contextualized Language and Knowledge Embedding (CoLAKE) CoLAKE jointly learns contextualized representation for both language and knowledge with the extended objective. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks.
  arXiv  Detail & Related papers  (2020-10-01T11:39:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.