Storing overlapping associative memories on latent manifolds in low-rank spiking networks

• URL: http://arxiv.org/abs/2411.17485v1
• Date: Tue, 26 Nov 2024 14:48:25 GMT
• Title: Storing overlapping associative memories on latent manifolds in low-rank spiking networks
• Authors: William F. Podlaski, Christian K. Machens,
• Abstract summary: We revisit the associative memory problem in light of advances in understanding spike-based computation. We show that the spiking activity for a large class of all-inhibitory networks is situated on a low-dimensional, convex, and piecewise-linear manifold. We propose several learning rules, and demonstrate a linear scaling of the storage capacity with the number of neurons, as well as robust pattern completion abilities.
• Score: 5.041384008847852
• License:
• Abstract: Associative memory architectures such as the Hopfield network have long been important conceptual and theoretical models for neuroscience and artificial intelligence. However, translating these abstract models into spiking neural networks has been surprisingly difficult. Indeed, much previous work has been restricted to storing a small number of primarily non-overlapping memories in large networks, thereby limiting their scalability. Here, we revisit the associative memory problem in light of recent advances in understanding spike-based computation. Using a recently-established geometric framework, we show that the spiking activity for a large class of all-inhibitory networks is situated on a low-dimensional, convex, and piecewise-linear manifold, with dynamics that move along the manifold. We then map the associative memory problem onto these dynamics, and demonstrate how the vertices of a hypercubic manifold can be used to store stable, overlapping activity patterns with a direct correspondence to the original Hopfield model. We propose several learning rules, and demonstrate a linear scaling of the storage capacity with the number of neurons, as well as robust pattern completion abilities. Overall, this work serves as a case study to demonstrate the effectiveness of using a geometrical perspective to design dynamics on neural manifolds, with implications for neuroscience and machine learning.

Related papers

• Hierarchical Working Memory and a New Magic Number [1.024113475677323]
  We propose a recurrent neural network model for chunking within the framework of the synaptic theory of working memory. Our work provides a novel conceptual and analytical framework for understanding the on-the-fly organization of information in the brain that is crucial for cognition.
  arXiv  Detail & Related papers  (2024-08-14T16:03:47Z)
• Explosive neural networks via higher-order interactions in curved statistical manifolds [43.496401697112695]
  We introduce curved neural networks as a class of prototypical models for studying higher-order phenomena. We show that these curved neural networks implement a self-regulating process that can accelerate memory retrieval.
  arXiv  Detail & Related papers  (2024-08-05T09:10:29Z)
• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Spiking representation learning for associative memories [0.0]
  We introduce a novel artificial spiking neural network (SNN) that performs unsupervised representation learning and associative memory operations. The architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories.
  arXiv  Detail & Related papers  (2024-06-05T08:30:11Z)
• A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes [49.32130498861987]
  We study the case of non-differentiable activation functions, such as ReLU. Two recent works introduced a geometric framework to study neural networks. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.
  arXiv  Detail & Related papers  (2024-04-09T08:11:46Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Hierarchical Associative Memory [2.66512000865131]
  Associative Memories or Modern Hopfield Networks have many appealing properties. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network. This paper tackles a gap and describes a fully recurrent model of associative memory with an arbitrary large number of layers.
  arXiv  Detail & Related papers  (2021-07-14T01:38:40Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)
• Large Associative Memory Problem in Neurobiology and Machine Learning [6.41804410246642]
  We present a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions.
  arXiv  Detail & Related papers  (2020-08-16T21:03:52Z)
• Hyperbolic Neural Networks++ [66.16106727715061]
  We generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincar'e ball model. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.
  arXiv  Detail & Related papers  (2020-06-15T08:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.