Knowledge Neurons in Pretrained Transformers

• URL: http://arxiv.org/abs/2104.08696v1
• Date: Sun, 18 Apr 2021 03:38:26 GMT
• Title: Knowledge Neurons in Pretrained Transformers
• Authors: Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Furu Wei
• Abstract summary: In this paper, we explore how implicit knowledge is stored in pretrained Transformers. We propose a knowledge attribution method to identify the neurons that express the fact. We show that the activation of such knowledge neurons is highly correlated to the expression of their corresponding facts.
• Score: 45.24499368763417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we explore how implicit knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons. Given a relational fact, we propose a knowledge attribution method to identify the neurons that express the fact. We present that the activation of such knowledge neurons is highly correlated to the expression of their corresponding facts. In addition, even without fine-tuning, we can leverage knowledge neurons to explicitly edit (such as update, and erase) specific factual knowledge for pretrained Transformers.

Related papers

• Knowledge Circuits in Pretrained Transformers [47.342682123081204]
  The inner workings of how modern large language models store knowledge have long been a subject of intense interest and investigation among researchers. In this paper, we delve into the graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. We evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies.
  arXiv  Detail & Related papers  (2024-05-28T08:56:33Z)
• Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons [20.56154830853632]
  This paper delves into the complex task of understanding how factual knowledge is stored in multilingual language models. We introduce the Architecture-adapted Multilingual Integrated Gradients method, which successfully localizes knowledge neurons more precisely. We also conduct an in-depth exploration of knowledge neurons, leading to the following two important discoveries.
  arXiv  Detail & Related papers  (2023-08-25T06:26:05Z)
• Constraints on the design of neuromorphic circuits set by the properties of neural population codes [61.15277741147157]
  In the brain, information is encoded, transmitted and used to inform behaviour. Neuromorphic circuits need to encode information in a way compatible to that used by populations of neuron in the brain.
  arXiv  Detail & Related papers  (2022-12-08T15:16:04Z)
• Finding Skill Neurons in Pre-trained Transformer-based Language Models [46.484656229427834]
  Transformer-based pre-trained language models have demonstrated superior performance on various natural language processing tasks. We find that after prompt tuning for specific tasks, the activations of some neurons within pre-trained Transformers are highly predictive of the task labels. We also explore the applications of skill neurons, including accelerating Transformers with network pruning and building better transferability indicators.
  arXiv  Detail & Related papers  (2022-11-14T13:43:46Z)
• Neural Knowledge Bank for Pretrained Transformers [20.416700112895974]
  We propose a Neural Knowledge Bank (NKB) to store extra factual knowledge for pretrained Transformers. During knowledge injection, we fix the original model and inject factual knowledge into the extended memory slots. We use three closed-book question answering datasets to show our strong ability to store extra factual knowledge.
  arXiv  Detail & Related papers  (2022-07-31T09:14:34Z)
• Finding patterns in Knowledge Attribution for Transformers [1.52292571922932]
  We use a 12-layer multi-lingual BERT model for our experiments. We observe that mostly factual knowledge can be attributed to middle and higher layers of the network. Applying the attribution scheme for grammatical knowledge, we find that grammatical knowledge is far more dispersed among the neurons than factual knowledge.
  arXiv  Detail & Related papers  (2022-05-03T08:30:51Z)
• Kformer: Knowledge Injection in Transformer Feed-Forward Layers [107.71576133833148]
  We propose a novel knowledge fusion model, namely Kformer, which incorporates external knowledge through the feed-forward layer in Transformer. We empirically find that simply injecting knowledge into FFN can facilitate the pre-trained language model's ability and facilitate current knowledge fusion methods.
  arXiv  Detail & Related papers  (2022-01-15T03:00:27Z)
• Towards a Universal Continuous Knowledge Base [49.95342223987143]
  We propose a method for building a continuous knowledge base that can store knowledge imported from multiple neural networks. Experiments on text classification show promising results. We import the knowledge from multiple models to the knowledge base, from which the fused knowledge is exported back to a single model.
  arXiv  Detail & Related papers  (2020-12-25T12:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.