A More Accurate Approximation of Activation Function with Few Spikes Neurons

• URL: http://arxiv.org/abs/2409.00044v1
• Date: Mon, 19 Aug 2024 02:08:56 GMT
• Title: A More Accurate Approximation of Activation Function with Few Spikes Neurons
• Authors: Dayena Jeong, Jaewoo Park, Jeonghee Jo, Jongkil Park, Jaewook Kim, Hyun Jae Jang, Suyoun Lee, Seongsik Park,
• Abstract summary: spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions.
• Score: 6.306126887439676
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent deep neural networks (DNNs), such as diffusion models [1], have faced high computational demands. Thus, spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. However, conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions, such as Swish [2]. To approximate activation functions with spiking neurons, few spikes (FS) neurons were proposed [3], but the approximation performance was limited due to the lack of training methods considering the neurons. Thus, we propose tendency-based parameter initialization (TBPI) to enhance the approximation of activation function with FS neurons, exploiting temporal dependencies initializing the training parameters.

Related papers

• Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
  Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks. In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
  arXiv  Detail & Related papers  (2024-09-17T04:48:45Z)
• Fast gradient-free activation maximization for neurons in spiking neural networks [5.805438104063613]
  We present a framework with an efficient design for such a loop. We track changes in the optimal stimuli for artificial neurons during training. This formation of refined optimal stimuli is associated with an increase in classification accuracy.
  arXiv  Detail & Related papers  (2023-12-28T18:30:13Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)
• Neural network with optimal neuron activation functions based on additive Gaussian process regression [0.0]
  More flexible neuron activation functions would allow using fewer neurons and layers and improve expressive power. We show that additive Gaussian process regression (GPR) can be used to construct optimal neuron activation functions that are individual to each neuron. An approach is also introduced that avoids non-linear fitting of neural network parameters.
  arXiv  Detail & Related papers  (2023-01-13T14:19:17Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Energy-Efficient High-Accuracy Spiking Neural Network Inference Using Time-Domain Neurons [0.18352113484137625]
  This paper presents a low-power highly linear time-domain I&F neuron circuit. The proposed neuron leads to more than 4.3x lower error rate on the MNIST inference. The power consumed by the proposed neuron circuit is simulated to be 0.230uW per neuron, which is orders of magnitude lower than the existing voltage-domain neurons.
  arXiv  Detail & Related papers  (2022-02-04T08:24:03Z)
• Improving Spiking Neural Network Accuracy Using Time-based Neurons [0.24366811507669117]
  Research on neuromorphic computing systems based on low-power spiking neural networks using analog neurons is in the spotlight. As technology scales down, analog neurons are difficult to scale, and they suffer from reduced voltage headroom/dynamic range and circuit nonlinearities. This paper first models the nonlinear behavior of existing current-mirror-based voltage-domain neurons designed in a 28nm process, and show SNN inference accuracy can be severely degraded by the effect of neuron's nonlinearity. We propose a novel neuron, which processes incoming spikes in the time domain and greatly improves the linearity, thereby improving the inference accuracy compared to the
  arXiv  Detail & Related papers  (2022-01-05T00:24:45Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks [0.9790524827475205]
  We show how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance. We calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks.
  arXiv  Detail & Related papers  (2020-05-24T01:04:53Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.