DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales

• URL: http://arxiv.org/abs/2104.04646v1
• Date: Fri, 9 Apr 2021 23:58:14 GMT
• Title: DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
• Authors: Brandon Jacques, Zoran Tiganj, Marc W. Howard, Per B. Sederberg
• Abstract summary: Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs) This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition -- and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves state-of-the-art performance on these problems.

Related papers

• Local Timescale Gates for Timescale-Robust Continual Spiking Neural Networks [0.17188280334580197]
  Spiking neural networks (SNNs) promise energy-efficient artificial intelligence on neuromorphic hardware.<n>We propose Local Timescale Gating (LT-Gate), a neuron model that combines dual time-constant dynamics with an adaptive gating mechanism.<n>Our results demonstrate that multi-timescale gating can substantially enhance continual learning in SNNs.
  arXiv  Detail & Related papers  (2025-10-13T23:31:07Z)
• Learning Delays Through Gradients and Structure: Emergence of Spatiotemporal Patterns in Spiking Neural Networks [0.06752396542927405]
  We present a Spiking Neural Network (SNN) model that incorporates learnable synaptic delays through two approaches. In the latter approach, the network selects and prunes connections, optimizing the delays in sparse connectivity settings. Our results demonstrate the potential of combining delay learning with dynamic pruning to develop efficient SNN models for temporal data processing.
  arXiv  Detail & Related papers  (2024-07-07T11:55:48Z)
• ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks [1.5931140598271163]
  We build a model for efficient learning sequences using only local always-on and phase-free plasticity. We showcase the capabilities of ELiSe in a mock-up of birdsong learning, and demonstrate its flexibility with respect to parametrization.
  arXiv  Detail & Related papers  (2024-02-26T17:30:34Z)
• TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling [54.97005925277638]
  The identification of sensory cues associated with potential opportunities and dangers is frequently complicated by unrelated events that separate useful cues by long delays. It remains a challenging task for state-of-the-art spiking neural networks (SNNs) to establish long-term temporal dependency between distant cues. We propose a novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF.
  arXiv  Detail & Related papers  (2023-08-25T08:54:41Z)
• Correlation-aware Spatial-Temporal Graph Learning for Multivariate Time-series Anomaly Detection [67.60791405198063]
  We propose a correlation-aware spatial-temporal graph learning (termed CST-GL) for time series anomaly detection. CST-GL explicitly captures the pairwise correlations via a multivariate time series correlation learning module. A novel anomaly scoring component is further integrated into CST-GL to estimate the degree of an anomaly in a purely unsupervised manner.
  arXiv  Detail & Related papers  (2023-07-17T11:04:27Z)
• Long Short-term Memory with Two-Compartment Spiking Neuron [64.02161577259426]
  We propose a novel biologically inspired Long Short-Term Memory Leaky Integrate-and-Fire spiking neuron model, dubbed LSTM-LIF. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, strong network generalizability, and high energy efficiency of the proposed LSTM-LIF model. This work, therefore, opens up a myriad of opportunities for resolving challenging temporal processing tasks on emerging neuromorphic computing machines.
  arXiv  Detail & Related papers  (2023-07-14T08:51:03Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Backpropagation with Biologically Plausible Spatio-Temporal Adjustment For Training Deep Spiking Neural Networks [5.484391472233163]
  The success of deep learning is inseparable from backpropagation. We propose a biological plausible spatial adjustment, which rethinks the relationship between membrane potential and spikes. Secondly, we propose a biologically plausible temporal adjustment making the error propagate across the spikes in the temporal dimension.
  arXiv  Detail & Related papers  (2021-10-17T15:55:51Z)
• An optimised deep spiking neural network architecture without gradients [7.183775638408429]
  We present an end-to-end trainable modular event-driven neural architecture that uses local synaptic and threshold adaptation rules. The architecture represents a highly abstracted model of existing Spiking Neural Network (SNN) architectures.
  arXiv  Detail & Related papers  (2021-09-27T05:59:12Z)
• Time-Reversal Symmetric ODE Network [138.02741983098454]
  Time-reversal symmetry is a fundamental property that frequently holds in classical and quantum mechanics. We propose a novel loss function that measures how well our ordinary differential equation (ODE) networks comply with this time-reversal symmetry. We show that, even for systems that do not possess the full time-reversal symmetry, TRS-ODENs can achieve better predictive performances over baselines.
  arXiv  Detail & Related papers  (2020-07-22T12:19:40Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)
• Depth Enables Long-Term Memory for Recurrent Neural Networks [0.0]
  We introduce a measure of the network's ability to support information flow across time, referred to as the Start-End separation rank. We prove that deep recurrent networks support Start-End separation ranks which are higher than those supported by their shallow counterparts.
  arXiv  Detail & Related papers  (2020-03-23T10:29:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.