Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering

• URL: http://arxiv.org/abs/2408.17322v1
• Date: Fri, 30 Aug 2024 14:32:25 GMT
• Title: Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering
• Authors: Nicholas Pochinkov, Ben Pasero, Skylar Shibayama,
• Abstract summary: We describe different lenses through which to view neuron activations, and investigate the effectiveness of language models and vision transformers. We find that in different regimes and models, each method can offer the lowest degradation of model performance compared to other methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of transformer-based models is growing rapidly throughout society. With this growth, it is important to understand how they work, and in particular, how the attention mechanisms represent concepts. Though there are many interpretability methods, many look at models through their neuronal activations, which are poorly understood. We describe different lenses through which to view neuron activations, and investigate the effectiveness in language models and vision transformers through various methods of neural ablation: zero ablation, mean ablation, activation resampling, and a novel approach we term 'peak ablation'. Through experimental analysis, we find that in different regimes and models, each method can offer the lowest degradation of model performance compared to other methods, with resampling usually causing the most significant performance deterioration. We make our code available at https://github.com/nickypro/investigating-ablation.

Related papers

• Single-neuron deep generative model uncovers underlying physics of neuronal activity in Ca imaging data [0.0]
  We propose a novel framework for single-neuron representation learning using autoregressive variational autoencoders (AVAEs) Our approach embeds individual neurons' signals into a reduced-dimensional space without the need for spike inference algorithms. The AVAE excels over traditional linear methods by generating more informative and discriminative latent representations.
  arXiv  Detail & Related papers  (2025-01-24T16:33:52Z)
• Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience [11.356550034255296]
  pruning techniques are widely used to remove a significant fraction of the network. These methods can reduce significant percent of the FLOPs, but often lead to a decrease in model performance. We propose a new concept for artificial neural network models named Neural Burden.
  arXiv  Detail & Related papers  (2024-07-23T03:43:21Z)
• Deep Latent Variable Modeling of Physiological Signals [0.8702432681310401]
  We explore high-dimensional problems related to physiological monitoring using latent variable models. First, we present a novel deep state-space model to generate electrical waveforms of the heart using optically obtained signals as inputs. Second, we present a brain signal modeling scheme that combines the strengths of probabilistic graphical models and deep adversarial learning. Third, we propose a framework for the joint modeling of physiological measures and behavior.
  arXiv  Detail & Related papers  (2024-05-29T17:07:33Z)
• Manipulating Feature Visualizations with Gradient Slingshots [54.31109240020007]
  We introduce a novel method for manipulating Feature Visualization (FV) without significantly impacting the model's decision-making process. We evaluate the effectiveness of our method on several neural network models and demonstrate its capabilities to hide the functionality of arbitrarily chosen neurons.
  arXiv  Detail & Related papers  (2024-01-11T18:57:17Z)
• Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding [82.46024259137823]
  We propose a cross-model comparative loss for a broad range of tasks. We demonstrate the universal effectiveness of comparative loss through extensive experiments on 14 datasets from 3 distinct NLU tasks.
  arXiv  Detail & Related papers  (2023-01-10T03:04:27Z)
• Generalization of generative model for neuronal ensemble inference method [0.0]
  In this study, we extend the range of the variable for expressing the neuronal state, and generalize the likelihood of the model for extended variables. This generalization without restriction of the binary input enables us to perform soft clustering and apply the method to non-stationary neuroactivity data.
  arXiv  Detail & Related papers  (2022-11-07T07:58:29Z)
• Learning Neural Causal Models with Active Interventions [83.44636110899742]
  We introduce an active intervention-targeting mechanism which enables a quick identification of the underlying causal structure of the data-generating process. Our method significantly reduces the required number of interactions compared with random intervention targeting. We demonstrate superior performance on multiple benchmarks from simulated to real-world data.
  arXiv  Detail & Related papers  (2021-09-06T13:10:37Z)
• On the Evolution of Neuron Communities in a Deep Learning Architecture [0.7106986689736827]
  This paper examines the neuron activation patterns of deep learning-based classification models. We show that both the community quality (modularity) and entropy are closely related to the deep learning models' performances.
  arXiv  Detail & Related papers  (2021-06-08T21:09:55Z)
• ACRE: Abstract Causal REasoning Beyond Covariation [90.99059920286484]
  We introduce the Abstract Causal REasoning dataset for systematic evaluation of current vision systems in causal induction. Motivated by the stream of research on causal discovery in Blicket experiments, we query a visual reasoning system with the following four types of questions in either an independent scenario or an interventional scenario. We notice that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning.
  arXiv  Detail & Related papers  (2021-03-26T02:42:38Z)
• Fooling the primate brain with minimal, targeted image manipulation [67.78919304747498]
  We propose an array of methods for creating minimal, targeted image perturbations that lead to changes in both neuronal activity and perception as reflected in behavior. Our work shares the same goal with adversarial attack, namely the manipulation of images with minimal, targeted noise that leads ANN models to misclassify the images.
  arXiv  Detail & Related papers  (2020-11-11T08:30:54Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)
• A new inference approach for training shallow and deep generalized linear models of noisy interacting neurons [4.899818550820575]
  We develop a two-step inference strategy that allows us to train robust generalized linear models of interacting neurons. We show that, compared to classical methods, the models trained in this way exhibit improved performance. The method can be extended to deep convolutional neural networks, leading to models with high predictive accuracy for both the neuron firing rates and their correlations.
  arXiv  Detail & Related papers  (2020-06-11T15:09:53Z)
• Towards Efficient Processing and Learning with Spikes: New Approaches for Multi-Spike Learning [59.249322621035056]
  We propose two new multi-spike learning rules which demonstrate better performance over other baselines on various tasks. In the feature detection task, we re-examine the ability of unsupervised STDP with its limitations being presented. Our proposed learning rules can reliably solve the task over a wide range of conditions without specific constraints being applied.
  arXiv  Detail & Related papers  (2020-05-02T06:41:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.