The Role of Temporal Hierarchy in Spiking Neural Networks

• URL: http://arxiv.org/abs/2407.18838v1
• Date: Fri, 26 Jul 2024 16:00:20 GMT
• Title: The Role of Temporal Hierarchy in Spiking Neural Networks
• Authors: Filippo Moro, Pau Vilimelis Aceituno, Laura Kriener, Melika Payvand,
• Abstract summary: Spiking Neural Networks (SNNs) have the potential for rich-temporal signal processing thanks to exploiting both spatial and temporal parameters. Time constants have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network. To reduce the cost of optimization, architectural biases can be applied, in this case in the temporal domain. We propose to impose a hierarchy of temporal representation in the hidden layers of SNNs, highlighting that such an inductive bias improves their performance.
• Score: 2.0881857682885836
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Spiking Neural Networks (SNNs) have the potential for rich spatio-temporal signal processing thanks to exploiting both spatial and temporal parameters. The temporal dynamics such as time constants of the synapses and neurons and delays have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network and increase the accuracy of the SNNs in solving temporal tasks. Optimizing such temporal parameters, for example, through gradient descent, gives rise to a temporal architecture for different problems. As has been shown in machine learning, to reduce the cost of optimization, architectural biases can be applied, in this case in the temporal domain. Such inductive biases in temporal parameters have been found in neuroscience studies, highlighting a hierarchy of temporal structure and input representation in different layers of the cortex. Motivated by this, we propose to impose a hierarchy of temporal representation in the hidden layers of SNNs, highlighting that such an inductive bias improves their performance. We demonstrate the positive effects of temporal hierarchy in the time constants of feed-forward SNNs applied to temporal tasks (Multi-Time-Scale XOR and Keyword Spotting, with a benefit of up to 4.1% in classification accuracy). Moreover, we show that such architectural biases, i.e. hierarchy of time constants, naturally emerge when optimizing the time constants through gradient descent, initialized as homogeneous values. We further pursue this proposal in temporal convolutional SNNs, by introducing the hierarchical bias in the size and dilation of temporal kernels, giving rise to competitive results in popular temporal spike-based datasets.

Related papers

• Towards Low-latency Event-based Visual Recognition with Hybrid Step-wise Distillation Spiking Neural Networks [50.32980443749865]
  Spiking neural networks (SNNs) have garnered significant attention for their low power consumption and high biologicalability. Current SNNs struggle to balance accuracy and latency in neuromorphic datasets. We propose Step-wise Distillation (HSD) method, tailored for neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-09-19T06:52:34Z)
• Temporal Reversed Training for Spiking Neural Networks with Generalized Spatio-Temporal Representation [3.5624857747396814]
  Spi neural networks (SNNs) have received widespread attention as an ultra-low energy computing paradigm. Recent studies have focused on improving the feature extraction capability of SNNs, but they suffer from inefficient and suboptimal performance. We. propose a simple yet effective temporal reversed training (TRT) method to optimize the temporal performance of SNNs.
  arXiv  Detail & Related papers  (2024-08-17T06:23:38Z)
• Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate [17.611912733951662]
  Recurrent Neural Networks (RNNs) are renowned for their adeptness in modeling temporal dependencies. We propose a novel Delayed Memory Unit (DMU) in this paper to enhance the temporal modeling capabilities of vanilla RNNs. Our proposed DMU demonstrates superior temporal modeling capabilities across a broad range of sequential modeling tasks.
  arXiv  Detail & Related papers  (2023-10-23T14:29:48Z)
• Time-Parameterized Convolutional Neural Networks for Irregularly Sampled Time Series [26.77596449192451]
  Irregularly sampled time series are ubiquitous in several application domains, leading to sparse, not fully-observed and non-aligned observations. Standard sequential neural networks (RNNs) and convolutional neural networks (CNNs) consider regular spacing between observation times, posing significant challenges to irregular time series modeling. We parameterize convolutional layers by employing time-explicitly irregular kernels.
  arXiv  Detail & Related papers  (2023-08-06T21:10:30Z)
• Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representation [67.26422477327179]
  Temporal graphs exhibit dynamic interactions between nodes over continuous time. We propose a novel method of temporal graph convolution with the whole neighborhood. Our proposed TAP-GNN outperforms existing temporal graph methods by a large margin in terms of both predictive performance and online inference latency.
  arXiv  Detail & Related papers  (2023-04-15T08:17:18Z)
• STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks [7.422913384086416]
  Spiking Neural Networks (SNNs), as one of the algorithmic models in neuromorphic computing, have gained a great deal of research attention owing to temporal processing capability. Existing synaptic structures in SNNs are almost full-connections or spatial 2D convolution, neither which can extract temporal dependencies adequately. We take inspiration from biological synapses and propose a synaptic connection SNN model, to enhance the synapse-temporal receptive fields of synaptic connections. We show that endowing synaptic models with temporal dependencies can improve the performance of SNNs on classification tasks.
  arXiv  Detail & Related papers  (2022-10-11T08:13:22Z)
• Scalable Spatiotemporal Graph Neural Networks [14.415967477487692]
  Graph neural networks (GNNs) are often the core component of the forecasting architecture. In most pretemporal GNNs, the computational complexity scales up to a quadratic factor with the length of the sequence times the number of links in the graph. We propose a scalable architecture that exploits an efficient encoding of both temporal and spatial dynamics.
  arXiv  Detail & Related papers  (2022-09-14T09:47:38Z)
• Space-Time Graph Neural Networks [104.55175325870195]
  We introduce space-time graph neural network (ST-GNN) to jointly process the underlying space-time topology of time-varying network data. Our analysis shows that small variations in the network topology and time evolution of a system does not significantly affect the performance of ST-GNNs.
  arXiv  Detail & Related papers  (2021-10-06T16:08:44Z)
• Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data with Spatial Information [52.635997570873194]
  This work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to process complex multi-dimensional time series data with spatial information. The proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.
  arXiv  Detail & Related papers  (2021-01-12T20:08:18Z)
• Multi-Tones' Phase Coding (MTPC) of Interaural Time Difference by Spiking Neural Network [68.43026108936029]
  We propose a pure spiking neural network (SNN) based computational model for precise sound localization in the noisy real-world environment. We implement this algorithm in a real-time robotic system with a microphone array. The experiment results show a mean error azimuth of 13 degrees, which surpasses the accuracy of the other biologically plausible neuromorphic approach for sound source localization.
  arXiv  Detail & Related papers  (2020-07-07T08:22:56Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.