Parallel Hyperparameter Optimization Of Spiking Neural Network
- URL: http://arxiv.org/abs/2403.00450v1
- Date: Fri, 1 Mar 2024 11:11:59 GMT
- Title: Parallel Hyperparameter Optimization Of Spiking Neural Network
- Authors: Thomas Firmin, Pierre Boulet, El-Ghazali Talbi
- Abstract summary: Spiking Neural Networks (SNNs) are based on a more biologically inspired approach than usual artificial neural networks.
We tackle the signal loss issue of SNNs to what we call silent networks.
By defining an early stopping criterion, we were able to instantiate larger and more flexible search spaces.
- Score: 0.5371337604556311
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired
approach than usual artificial neural networks. Such models are characterized
by complex dynamics between neurons and spikes. These are very sensitive to the
hyperparameters, making their optimization challenging. To tackle
hyperparameter optimization of SNNs, we initially extended the signal loss
issue of SNNs to what we call silent networks. These networks fail to emit
enough spikes at their outputs due to mistuned hyperparameters or architecture.
Generally, search spaces are heavily restrained, sometimes even discretized, to
prevent the sampling of such networks. By defining an early stopping criterion
detecting silent networks and by designing specific constraints, we were able
to instantiate larger and more flexible search spaces. We applied a constrained
Bayesian optimization technique, which was asynchronously parallelized, as the
evaluation time of a SNN is highly stochastic. Large-scale experiments were
carried-out on a multi-GPU Petascale architecture. By leveraging silent
networks, results show an acceleration of the search, while maintaining good
performances of both the optimization algorithm and the best solution obtained.
We were able to apply our methodology to two popular training algorithms, known
as spike timing dependent plasticity and surrogate gradient. Early detection
allowed us to prevent worthless and costly computation, directing the search
toward promising hyperparameter combinations. Our methodology could be applied
to multi-objective problems, where the spiking activity is often minimized to
reduce the energy consumption. In this scenario, it becomes essential to find
the delicate frontier between low-spiking and silent networks. Finally, our
approach may have implications for neural architecture search, particularly in
defining suitable spiking architectures.
Related papers
- LitE-SNN: Designing Lightweight and Efficient Spiking Neural Network through Spatial-Temporal Compressive Network Search and Joint Optimization [48.41286573672824]
Spiking Neural Networks (SNNs) mimic the information-processing mechanisms of the human brain and are highly energy-efficient.
We propose a new approach named LitE-SNN that incorporates both spatial and temporal compression into the automated network design process.
arXiv Detail & Related papers (2024-01-26T05:23:11Z) - Low Latency of object detection for spikng neural network [3.404826786562694]
Spiking Neural Networks are well-suited for edge AI applications due to their binary spike nature.
In this paper, we focus on generating highly accurate and low-latency SNNs specifically for object detection.
arXiv Detail & Related papers (2023-09-27T10:26:19Z) - Inherent Redundancy in Spiking Neural Networks [24.114844269113746]
Spiking Networks (SNNs) are a promising energy-efficient alternative to conventional artificial neural networks.
In this work, we focus on three key questions regarding inherent redundancy in SNNs.
We propose an Advance Attention (ASA) module to harness SNNs' redundancy.
arXiv Detail & Related papers (2023-08-16T08:58:25Z) - Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency.
We present an IPU-optimized release of our custom SNN Python package, snnTorch.
arXiv Detail & Related papers (2022-11-19T15:44:08Z) - Bayesian Hyperparameter Optimization for Deep Neural Network-Based
Network Intrusion Detection [2.304713283039168]
Deep neural networks (DNN) have been successfully applied for intrusion detection problems.
This paper proposes a novel Bayesian optimization-based framework for the automatic optimization of hyper parameters.
We show that the proposed framework demonstrates significantly higher intrusion detection performance than the random search optimization-based approach.
arXiv Detail & Related papers (2022-07-07T20:08:38Z) - Training High-Performance Low-Latency Spiking Neural Networks by
Differentiation on Spike Representation [70.75043144299168]
Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware.
It is a challenge to efficiently train SNNs due to their non-differentiability.
We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
arXiv Detail & Related papers (2022-05-01T12:44:49Z) - Comparative Analysis of Interval Reachability for Robust Implicit and
Feedforward Neural Networks [64.23331120621118]
We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs)
INNs are a class of implicit learning models that use implicit equations as layers.
We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
arXiv Detail & Related papers (2022-04-01T03:31:27Z) - Navigating Local Minima in Quantized Spiking Neural Networks [3.1351527202068445]
Spiking and Quantized Neural Networks (NNs) are becoming exceedingly important for hyper-efficient implementations of Deep Learning (DL) algorithms.
These networks face challenges when trained using error backpropagation, due to the absence of gradient signals when applying hard thresholds.
This paper presents a systematic evaluation of a cosine-annealed LR schedule coupled with weight-independent adaptive moment estimation.
arXiv Detail & Related papers (2022-02-15T06:42:25Z) - Encoding the latent posterior of Bayesian Neural Networks for
uncertainty quantification [10.727102755903616]
We aim for efficient deep BNNs amenable to complex computer vision architectures.
We achieve this by leveraging variational autoencoders (VAEs) to learn the interaction and the latent distribution of the parameters at each network layer.
Our approach, Latent-Posterior BNN (LP-BNN), is compatible with the recent BatchEnsemble method, leading to highly efficient (in terms of computation and memory during both training and testing) ensembles.
arXiv Detail & Related papers (2020-12-04T19:50:09Z) - Rectified Linear Postsynaptic Potential Function for Backpropagation in
Deep Spiking Neural Networks [55.0627904986664]
Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation.
This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
arXiv Detail & Related papers (2020-03-26T11:13:07Z) - MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks.
The use of gradient combined nonvolutionity renders learning susceptible to novel problems.
We propose fusing neighboring layers of deeper networks that are trained with random variables.
arXiv Detail & Related papers (2020-01-28T18:25:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.