Slow manifolds in recurrent networks encode working memory efficiently and robustly

• URL: http://arxiv.org/abs/2101.03163v1
• Date: Fri, 8 Jan 2021 18:47:02 GMT
• Title: Slow manifolds in recurrent networks encode working memory efficiently and robustly
• Authors: Elham Ghazizadeh, ShiNung Ching
• Abstract summary: Working memory is a cognitive function involving the storage and manipulation of latent information over brief intervals of time. We use a top-down modeling approach to examine network-level mechanisms of working memory.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Working memory is a cognitive function involving the storage and manipulation of latent information over brief intervals of time, thus making it crucial for context-dependent computation. Here, we use a top-down modeling approach to examine network-level mechanisms of working memory, an enigmatic issue and central topic of study in neuroscience and machine intelligence. We train thousands of recurrent neural networks on a working memory task and then perform dynamical systems analysis on the ensuing optimized networks, wherein we find that four distinct dynamical mechanisms can emerge. In particular, we show the prevalence of a mechanism in which memories are encoded along slow stable manifolds in the network state space, leading to a phasic neuronal activation profile during memory periods. In contrast to mechanisms in which memories are directly encoded at stable attractors, these networks naturally forget stimuli over time. Despite this seeming functional disadvantage, they are more efficient in terms of how they leverage their attractor landscape and paradoxically, are considerably more robust to noise. Our results provide new dynamical hypotheses regarding how working memory function is encoded in both natural and artificial neural networks.

Related papers

• Hierarchical Working Memory and a New Magic Number [1.024113475677323]
  We propose a recurrent neural network model for chunking within the framework of the synaptic theory of working memory. Our work provides a novel conceptual and analytical framework for understanding the on-the-fly organization of information in the brain that is crucial for cognition.
  arXiv  Detail & Related papers  (2024-08-14T16:03:47Z)
• Spiking representation learning for associative memories [0.0]
  We introduce a novel artificial spiking neural network (SNN) that performs unsupervised representation learning and associative memory operations. The architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories.
  arXiv  Detail & Related papers  (2024-06-05T08:30:11Z)
• In search of dispersed memories: Generative diffusion models are associative memory networks [6.4322891559626125]
  Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. We show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network.
  arXiv  Detail & Related papers  (2023-09-29T14:48:24Z)
• Long Short-term Memory with Two-Compartment Spiking Neuron [64.02161577259426]
  We propose a novel biologically inspired Long Short-Term Memory Leaky Integrate-and-Fire spiking neuron model, dubbed LSTM-LIF. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, strong network generalizability, and high energy efficiency of the proposed LSTM-LIF model. This work, therefore, opens up a myriad of opportunities for resolving challenging temporal processing tasks on emerging neuromorphic computing machines.
  arXiv  Detail & Related papers  (2023-07-14T08:51:03Z)
• Sparse Coding in a Dual Memory System for Lifelong Learning [13.041607703862724]
  Brain efficiently encodes information in non-overlapping sparse codes. We employ sparse coding in a multiple-memory replay mechanism. Our method maintains an additional long-term semantic memory that aggregates and consolidates information encoded in the synaptic weights of the working model.
  arXiv  Detail & Related papers  (2022-12-28T12:56:15Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Cross-Frequency Coupling Increases Memory Capacity in Oscillatory Neural Networks [69.42260428921436]
  Cross-frequency coupling (CFC) is associated with information integration across populations of neurons. We construct a model of CFC which predicts a computational role for observed $theta - gamma$ oscillatory circuits in the hippocampus and cortex. We show that the presence of CFC increases the memory capacity of a population of neurons connected by plastic synapses.
  arXiv  Detail & Related papers  (2022-04-05T17:13:36Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Neuromorphic Algorithm-hardware Codesign for Temporal Pattern Learning [11.781094547718595]
  We derive an efficient training algorithm for Leaky Integrate and Fire neurons, which is capable of training a SNN to learn complex spatial temporal patterns. We have developed a CMOS circuit implementation for a memristor-based network of neuron and synapses which retains critical neural dynamics with reduced complexity.
  arXiv  Detail & Related papers  (2021-04-21T18:23:31Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)
• Encoding-based Memory Modules for Recurrent Neural Networks [79.42778415729475]
  We study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences.
  arXiv  Detail & Related papers  (2020-01-31T11:14:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.