NeuroX Library for Neuron Analysis of Deep NLP Models
- URL: http://arxiv.org/abs/2305.17073v1
- Date: Fri, 26 May 2023 16:32:56 GMT
- Title: NeuroX Library for Neuron Analysis of Deep NLP Models
- Authors: Fahim Dalvi and Hassan Sajjad and Nadir Durrani
- Abstract summary: We present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis of natural language processing models.
NeuroX implements various interpretation methods under a unified API, and provides a framework for data processing and evaluation.
- Score: 21.663464746974455
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Neuron analysis provides insights into how knowledge is structured in
representations and discovers the role of neurons in the network. In addition
to developing an understanding of our models, neuron analysis enables various
applications such as debiasing, domain adaptation and architectural search. We
present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis
of natural language processing models. It implements various interpretation
methods under a unified API, and provides a framework for data processing and
evaluation, thus making it easier for researchers and practitioners to perform
neuron analysis. The Python toolkit is available at
https://www.github.com/fdalvi/NeuroX. Demo Video available at
https://youtu.be/mLhs2YMx4u8.
Related papers
- Graph Neural Networks for Brain Graph Learning: A Survey [53.74244221027981]
Graph neural networks (GNNs) have demonstrated a significant advantage in mining graph-structured data.
GNNs to learn brain graph representations for brain disorder analysis has recently gained increasing attention.
In this paper, we aim to bridge this gap by reviewing brain graph learning works that utilize GNNs.
arXiv Detail & Related papers (2024-06-01T02:47:39Z) - Neuron-Level Knowledge Attribution in Large Language Models [19.472889262384818]
We propose a method for pinpointing significant neurons for different outputs.
Our approach demonstrates superior performance across three metrics.
We will release our data and code on storage and set the stage for future research in knowledge editing.
arXiv Detail & Related papers (2023-12-19T13:23:18Z) - WaLiN-GUI: a graphical and auditory tool for neuron-based encoding [73.88751967207419]
Neuromorphic computing relies on spike-based, energy-efficient communication.
We develop a tool to identify suitable configurations for neuron-based encoding of sample-based data into spike trains.
The WaLiN-GUI is provided open source and with documentation.
arXiv Detail & Related papers (2023-10-25T20:34:08Z) - Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey) [9.14580723964253]
Can we obtain insights about the brain using AI models?
How is the information in deep learning models related to brain recordings?
Decoding models solve the inverse problem of reconstructing stimuli given the fMRI.
Inspired by the effectiveness of deep learning models for natural language processing, computer vision, and speech, several neural encoding and decoding models have been recently proposed.
arXiv Detail & Related papers (2023-07-17T06:54:36Z) - Neuron to Graph: Interpreting Language Model Neurons at Scale [8.32093320910416]
This paper introduces a novel automated approach designed to scale interpretability techniques across a vast array of neurons within Large Language Models.
We propose Neuron to Graph (N2G), an innovative tool that automatically extracts a neuron's behaviour from the dataset it was trained on and translates it into an interpretable graph.
arXiv Detail & Related papers (2023-05-31T14:44:33Z) - N2G: A Scalable Approach for Quantifying Interpretable Neuron
Representations in Large Language Models [0.0]
N2G is a tool which takes a neuron and its dataset examples, and automatically distills the neuron's behaviour on those examples to an interpretable graph.
We use truncation and saliency methods to only present the important tokens, and augment the dataset examples with more diverse samples to better capture the extent of neuron behaviour.
These graphs can be visualised to aid manual interpretation by researchers, but can also output token activations on text to compare to the neuron's ground truth activations for automatic validation.
arXiv Detail & Related papers (2023-04-22T19:06:13Z) - Neuromorphic Artificial Intelligence Systems [58.1806704582023]
Modern AI systems, based on von Neumann architecture and classical neural networks, have a number of fundamental limitations in comparison with the brain.
This article discusses such limitations and the ways they can be mitigated.
It presents an overview of currently available neuromorphic AI projects in which these limitations are overcome.
arXiv Detail & Related papers (2022-05-25T20:16:05Z) - The Neural Coding Framework for Learning Generative Models [91.0357317238509]
We propose a novel neural generative model inspired by the theory of predictive processing in the brain.
In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
arXiv Detail & Related papers (2020-12-07T01:20:38Z) - Compositional Explanations of Neurons [52.71742655312625]
We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts.
We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
arXiv Detail & Related papers (2020-06-24T20:37:05Z) - Artificial neural networks for neuroscientists: A primer [4.771833920251869]
Artificial neural networks (ANNs) are essential tools in machine learning that have drawn increasing attention in neuroscience.
In this pedagogical Primer, we introduce ANNs and demonstrate how they have been fruitfully deployed to study neuroscientific questions.
With a focus on bringing this mathematical framework closer to neurobiology, we detail how to customize the analysis, structure, and learning of ANNs.
arXiv Detail & Related papers (2020-06-01T15:08:42Z) - Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy.
We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
arXiv Detail & Related papers (2020-02-02T21:09:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.