Exploring weight initialization, diversity of solutions, and degradation in recurrent neural networks trained for temporal and decision-making tasks

• URL: http://arxiv.org/abs/1906.01094v6
• Date: Wed, 28 Jun 2023 07:52:10 GMT
• Title: Exploring weight initialization, diversity of solutions, and degradation in recurrent neural networks trained for temporal and decision-making tasks
• Authors: Cecilia Jarne and Rodrigo Laje
• Abstract summary: Recurrent Neural Networks (RNNs) are frequently used to model aspects of brain function and structure. In this work, we trained small fully-connected RNNs to perform temporal and flow control tasks with time-varying stimuli.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recurrent Neural Networks (RNNs) are frequently used to model aspects of brain function and structure. In this work, we trained small fully-connected RNNs to perform temporal and flow control tasks with time-varying stimuli. Our results show that different RNNs can solve the same task by converging to different underlying dynamics and also how the performance gracefully degrades as either network size is decreased, interval duration is increased, or connectivity damage is increased. For the considered tasks, we explored how robust the network obtained after training can be according to task parameterization. In the process, we developed a framework that can be useful to parameterize other tasks of interest in computational neuroscience. Our results are useful to quantify different aspects of the models, which are normally used as black boxes and need to be understood in order to model the biological response of cerebral cortex areas.

Related papers

• Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks [2.184775414778289]
  We provide a unified framework for analyzing degeneracy across three levels: behavior, neural dynamics, and weight space. We analyzed RNNs trained on diverse tasks across machine learning and neuroscience domains.
  arXiv  Detail & Related papers  (2024-10-04T23:23:55Z)
• Context Gating in Spiking Neural Networks: Achieving Lifelong Learning through Integration of Local and Global Plasticity [20.589970453110208]
  Humans learn multiple tasks in succession with minimal mutual interference, through the context gating mechanism in the prefrontal cortex (PFC) We propose SNN with context gating trained by the local plasticity rule (CG-SNN) for lifelong learning. Experiments show that the proposed model is effective in maintaining the past learning experience and has better task-selectivity than other methods during lifelong learning.
  arXiv  Detail & Related papers  (2024-06-04T01:35:35Z)
• Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks [9.889775504641925]
  We propose a brain-inspired continual learning algorithm with adaptive reorganization of neural pathways. The proposed model demonstrates consistent superiority in performance, energy consumption, and memory capacity on diverse continual learning tasks.
  arXiv  Detail & Related papers  (2023-09-18T07:56:40Z)
• Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks [6.407825206595442]
  Children possess the ability to learn multiple cognitive tasks sequentially. Existing continual learning frameworks are usually applicable to Deep Neural Networks (DNNs) We propose Dynamic Structure Development of Spiking Neural Networks (DSD-SNN) for efficient and adaptive continual learning.
  arXiv  Detail & Related papers  (2023-08-09T07:36:40Z)
• Long Short-term Memory with Two-Compartment Spiking Neuron [64.02161577259426]
  We propose a novel biologically inspired Long Short-Term Memory Leaky Integrate-and-Fire spiking neuron model, dubbed LSTM-LIF. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, strong network generalizability, and high energy efficiency of the proposed LSTM-LIF model. This work, therefore, opens up a myriad of opportunities for resolving challenging temporal processing tasks on emerging neuromorphic computing machines.
  arXiv  Detail & Related papers  (2023-07-14T08:51:03Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks [0.9790524827475205]
  We show how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance. We calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks.
  arXiv  Detail & Related papers  (2020-05-24T01:04:53Z)
• Recurrent Neural Network Learning of Performance and Intrinsic Population Dynamics from Sparse Neural Data [77.92736596690297]
  We introduce a novel training strategy that allows learning not only the input-output behavior of an RNN but also its internal network dynamics. We test the proposed method by training an RNN to simultaneously reproduce internal dynamics and output signals of a physiologically-inspired neural model. Remarkably, we show that the reproduction of the internal dynamics is successful even when the training algorithm relies on the activities of a small subset of neurons.
  arXiv  Detail & Related papers  (2020-05-05T14:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.