Identification of Knowledge Neurons in Protein Language Models

• URL: http://arxiv.org/abs/2312.10770v1
• Date: Sun, 17 Dec 2023 17:23:43 GMT
• Title: Identification of Knowledge Neurons in Protein Language Models
• Authors: Divya Nori, Shivali Singireddy, Marina Ten Have
• Abstract summary: We identify and characterizing knowledge neurons, components that express understanding of key information. We show that there is a high density of knowledge neurons in the key vector prediction networks of self-attention modules. In the future, the types of knowledge captured by each neuron could be characterized.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural language models have become powerful tools for learning complex representations of entities in natural language processing tasks. However, their interpretability remains a significant challenge, particularly in domains like computational biology where trust in model predictions is crucial. In this work, we aim to enhance the interpretability of protein language models, specifically the state-of-the-art ESM model, by identifying and characterizing knowledge neurons - components that express understanding of key information. After fine-tuning the ESM model for the task of enzyme sequence classification, we compare two knowledge neuron selection methods that preserve a subset of neurons from the original model. The two methods, activation-based and integrated gradient-based selection, consistently outperform a random baseline. In particular, these methods show that there is a high density of knowledge neurons in the key vector prediction networks of self-attention modules. Given that key vectors specialize in understanding different features of input sequences, these knowledge neurons could capture knowledge of different enzyme sequence motifs. In the future, the types of knowledge captured by each neuron could be characterized.

Related papers

• Modularity in Transformers: Investigating Neuron Separability & Specialization [0.0]
  Transformer models are increasingly prevalent in various applications, yet our understanding of their internal workings remains limited. This paper investigates the modularity and task specialization of neurons within transformer architectures, focusing on both vision (ViT) and language (Mistral 7B) models. Using a combination of selective pruning and MoEfication clustering techniques, we analyze the overlap and specialization of neurons across different tasks and data subsets.
  arXiv  Detail & Related papers  (2024-08-30T14:35:01Z)
• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment [65.268245109828]
  We examine algorithms for conducting credit assignment in artificial neural networks that are inspired or motivated by neurobiology. We organize the ever-growing set of brain-inspired learning schemes into six general families and consider these in the context of backpropagation of errors. The results of this review are meant to encourage future developments in neuro-mimetic systems and their constituent learning processes.
  arXiv  Detail & Related papers  (2023-12-01T05:20:57Z)
• Investigating the Encoding of Words in BERT's Neurons using Feature Textualization [11.943486282441143]
  We propose a technique to produce representations of neurons in embedding word space. We find that the produced representations can provide insights about the encoded knowledge in individual neurons.
  arXiv  Detail & Related papers  (2023-11-14T15:21:49Z)
• Discovering Salient Neurons in Deep NLP Models [31.18937787704794]
  We present a technique called as Linguistic Correlation Analysis to extract salient neurons in the model. Our data-driven, quantitative analysis illuminates interesting findings. Our code is publicly available as part of the NeuroX toolkit.
  arXiv  Detail & Related papers  (2022-06-27T13:31:49Z)
• Dependency-based Mixture Language Models [53.152011258252315]
  We introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
  arXiv  Detail & Related papers  (2022-03-19T06:28:30Z)
• Natural Language Descriptions of Deep Visual Features [50.270035018478666]
  We introduce a procedure that automatically labels neurons with open-ended, compositional, natural language descriptions. We use MILAN for analysis, characterizing the distribution and importance of neurons selective for attribute, category, and relational information in vision models. We also use MILAN for auditing, surfacing neurons sensitive to protected categories like race and gender in models trained on datasets intended to obscure these features.
  arXiv  Detail & Related papers  (2022-01-26T18:48:02Z)
• Self-Supervised Graph Representation Learning for Neuronal Morphologies [75.38832711445421]
  We present GraphDINO, a data-driven approach to learn low-dimensional representations of 3D neuronal morphologies from unlabeled datasets. We show, in two different species and across multiple brain areas, that this method yields morphological cell type clusterings on par with manual feature-based classification by experts. Our method could potentially enable data-driven discovery of novel morphological features and cell types in large-scale datasets.
  arXiv  Detail & Related papers  (2021-12-23T12:17:47Z)
• Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE [10.529943544385585]
  We propose a method that integrates key ingredients from latent models and traditional neural encoding models. Our method, pi-VAE, is inspired by recent progress on identifiable variational auto-encoder. We validate pi-VAE using synthetic data, and apply it to analyze neurophysiological datasets from rat hippocampus and macaque motor cortex.
  arXiv  Detail & Related papers  (2020-11-09T22:00:38Z)
• Analyzing Individual Neurons in Pre-trained Language Models [41.07850306314594]
  We find small subsets of neurons to predict linguistic tasks, with lower level tasks localized in fewer neurons, compared to higher level task of predicting syntax. For example, we found neurons in XLNet to be more localized and disjoint when predicting properties compared to BERT and others, where they are more distributed and coupled.
  arXiv  Detail & Related papers  (2020-10-06T13:17:38Z)
• Compositional Explanations of Neurons [52.71742655312625]
  We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts. We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
  arXiv  Detail & Related papers  (2020-06-24T20:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.