Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks

• URL: http://arxiv.org/abs/2105.05457v1
• Date: Wed, 12 May 2021 06:45:26 GMT
• Title: Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks
• Authors: Ting-Yun Chang, Yang Liu, Karthik Gopalakrishnan, Behnam Hedayatnia, Pei Zhou, Dilek Hakkani-Tur
• Abstract summary: External commonsense knowledge graphs (KGs) provide rich information about words and their relationships. We propose two approaches to emphimplicitly and emphexplicitly infuse such KGs into pretrained language models. We demonstrate our proposed methods perform well on SocialIQA, a social commonsense reasoning task, in both limited and full training data.
• Score: 6.335245542129822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained language models have excelled at many NLP tasks recently; however, their social intelligence is still unsatisfactory. To enable this, machines need to have a more general understanding of our complicated world and develop the ability to perform commonsense reasoning besides fitting the specific downstream tasks. External commonsense knowledge graphs (KGs), such as ConceptNet, provide rich information about words and their relationships. Thus, towards general commonsense learning, we propose two approaches to \emph{implicitly} and \emph{explicitly} infuse such KGs into pretrained language models. We demonstrate our proposed methods perform well on SocialIQA, a social commonsense reasoning task, in both limited and full training data regimes.

Related papers

• Commonsense Knowledge Transfer for Pre-trained Language Models [83.01121484432801]
  We introduce commonsense knowledge transfer, a framework to transfer the commonsense knowledge stored in a neural commonsense knowledge model to a general-purpose pre-trained language model. It first exploits general texts to form queries for extracting commonsense knowledge from the neural commonsense knowledge model. It then refines the language model with two self-supervised objectives: commonsense mask infilling and commonsense relation prediction.
  arXiv  Detail & Related papers  (2023-06-04T15:44:51Z)
• LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning [12.278877764015725]
  We present LaSQuE, a method that can learn zero-shot classifiers from language explanations by using three new strategies. With these strategies, LaSQuE outperforms prior work, showing an absolute gain of up to 7% in generalizing to unseen real-world classification tasks.
  arXiv  Detail & Related papers  (2022-12-18T15:10:05Z)
• DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning [18.838291575019504]
  Pre-trained language models (PLMs) are shown to be lacking in knowledge when dealing with knowledge driven tasks. We propose textbfDictBERT, a novel approach that enhances PLMs with dictionary knowledge. We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE.
  arXiv  Detail & Related papers  (2022-08-01T06:43:19Z)
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering [159.9645181522436]
  GreaseLM is a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. We show that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
  arXiv  Detail & Related papers  (2022-01-21T19:00:05Z)
• Generated Knowledge Prompting for Commonsense Reasoning [53.88983683513114]
  We propose generating knowledge statements directly from a language model with a generic prompt format. This approach improves performance of both off-the-shelf and finetuned language models on four commonsense reasoning tasks. Notably, we find that a model's predictions can improve when using its own generated knowledge.
  arXiv  Detail & Related papers  (2021-10-15T21:58:03Z)
• Understanding Few-Shot Commonsense Knowledge Models [39.31365020474205]
  We investigate training commonsense knowledge models in a few-shot setting. We find that human quality ratings for knowledge produced from a few-shot trained system can achieve performance within 6% of knowledge produced from fully supervised systems.
  arXiv  Detail & Related papers  (2021-01-01T19:01:09Z)
• CoCoLM: COmplex COmmonsense Enhanced Language Model [45.396629052897524]
  We propose to help pre-trained language models better incorporate complex commonsense knowledge. Different from existing fine-tuning approaches, we do not focus on a specific task and propose a general language model named CoCoLM. We successfully teach pre-trained language models rich complex commonsense knowledge among eventualities.
  arXiv  Detail & Related papers  (2020-12-31T15:05:36Z)
• JAKET: Joint Pre-training of Knowledge Graph and Language Understanding [73.43768772121985]
  We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains.
  arXiv  Detail & Related papers  (2020-10-02T05:53:36Z)
• Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge [96.92252296244233]
  Large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. We show that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
  arXiv  Detail & Related papers  (2020-06-11T17:02:20Z)
• Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers [54.417299589288184]
  We investigate models for complementing the distributional knowledge of BERT with conceptual knowledge from ConceptNet and its corresponding Open Mind Common Sense (OMCS) corpus. Our adapter-based models substantially outperform BERT on inference tasks that require the type of conceptual knowledge explicitly present in ConceptNet and OMCS.
  arXiv  Detail & Related papers  (2020-05-24T15:49:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.