Fast gradient-free activation maximization for neurons in spiking neural networks

• URL: http://arxiv.org/abs/2401.10748v2
• Date: Tue, 25 Jun 2024 17:08:56 GMT
• Title: Fast gradient-free activation maximization for neurons in spiking neural networks
• Authors: Nikita Pospelov, Andrei Chertkov, Maxim Beketov, Ivan Oseledets, Konstantin Anokhin,
• Abstract summary: We present a framework with an efficient design for such a loop. We track changes in the optimal stimuli for artificial neurons during training. This formation of refined optimal stimuli is associated with an increase in classification accuracy.
• Score: 5.805438104063613
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Elements of neural networks, both biological and artificial, can be described by their selectivity for specific cognitive features. Understanding these features is important for understanding the inner workings of neural networks. For a living system, such as a neuron, whose response to a stimulus is unknown and not differentiable, the only way to reveal these features is through a feedback loop that exposes it to a large set of different stimuli. The properties of these stimuli should be varied iteratively in order to maximize the neuronal response. To utilize this feedback loop for a biological neural network, it is important to run it quickly and efficiently in order to reach the stimuli that maximizes certain neurons' activation with the least number of iterations possible. Here we present a framework with an efficient design for such a loop. We successfully tested it on an artificial spiking neural network (SNN), which is a model that simulates the asynchronous spiking activity of neurons in living brains. Our optimization method for activation maximization is based on the low-rank Tensor Train decomposition of the discrete activation function. The optimization space is the latent parameter space of images generated by SN-GAN or VQ-VAE generative models. To our knowledge, this is the first time that effective AM has been applied to SNNs. We track changes in the optimal stimuli for artificial neurons during training and show that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network. This formation of refined optimal stimuli is associated with an increase in classification accuracy. Some neurons, especially in the deeper layers, may gradually change the concepts they are selective for during learning, potentially explaining their importance for model performance.

Related papers

• To Spike or Not to Spike, that is the Question [0.45609532372046985]
  SNNs emulate the temporal dynamics of biological neurons, making them well-suited for real-time, event-driven processing. In SNNs, learning rules are based on neurons' spiking behavior, that is, if and when spikes are generated due to a neuron's membrane potential exceeding that neuron's spiking threshold. This work focuses on the significance of neuron learning thresholds alongside weights in SNNs.
  arXiv  Detail & Related papers  (2024-07-28T19:23:09Z)
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks [74.3099028063756]
  We develop a new method with neuronal operations based on lateral connections and Hebbian learning. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities. Our method consistently solves for spiking neural networks with nearly zero forgetting.
  arXiv  Detail & Related papers  (2024-02-19T09:29:37Z)
• Neuroscience inspired scientific machine learning (Part-1): Variable spiking neuron for regression [2.1756081703276]
  We introduce in this paper a novel spiking neuron, termed Variable Spiking Neuron (VSN) It can reduce the redundant firing using lessons from biological neuron inspired Leaky Integrate and Fire Spiking Neurons (LIF-SN)
  arXiv  Detail & Related papers  (2023-11-15T08:59:06Z)
• Learning to Act through Evolution of Neural Diversity in Random Neural Networks [9.387749254963595]
  In most artificial neural networks (ANNs), neural computation is abstracted to an activation function that is usually shared between all neurons. We propose the optimization of neuro-centric parameters to attain a set of diverse neurons that can perform complex computations.
  arXiv  Detail & Related papers  (2023-05-25T11:33:04Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Biologically-inspired neuronal adaptation improves learning in neural networks [0.7734726150561086]
  Humans still outperform artificial neural networks on many tasks. We draw inspiration from the brain to improve machine learning algorithms. We add adaptation to multilayer perceptrons and convolutional neural networks trained on MNIST and CIFAR-10.
  arXiv  Detail & Related papers  (2022-04-08T16:16:02Z)
• Event-based Video Reconstruction via Potential-assisted Spiking Neural Network [48.88510552931186]
  Bio-inspired neural networks can potentially lead to greater computational efficiency on event-driven hardware. We propose a novel Event-based Video reconstruction framework based on a fully Spiking Neural Network (EVSNN) We find that the spiking neurons have the potential to store useful temporal information (memory) to complete such time-dependent tasks.
  arXiv  Detail & Related papers  (2022-01-25T02:05:20Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks [0.9790524827475205]
  We show how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance. We calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks.
  arXiv  Detail & Related papers  (2020-05-24T01:04:53Z)
• Recurrent Neural Network Learning of Performance and Intrinsic Population Dynamics from Sparse Neural Data [77.92736596690297]
  We introduce a novel training strategy that allows learning not only the input-output behavior of an RNN but also its internal network dynamics. We test the proposed method by training an RNN to simultaneously reproduce internal dynamics and output signals of a physiologically-inspired neural model. Remarkably, we show that the reproduction of the internal dynamics is successful even when the training algorithm relies on the activities of a small subset of neurons.
  arXiv  Detail & Related papers  (2020-05-05T14:16:54Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.