Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

• URL: http://arxiv.org/abs/2202.04557v1
• Date: Wed, 9 Feb 2022 16:48:06 GMT
• Title: Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models
• Authors: Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, Rafal Bogacz
• Abstract summary: We propose a general framework for understanding the operation of memory networks as a sequence of three operations. We derive all these memory models as instances of our general framework with differing similarity and separation functions.
• Score: 41.58529335439799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov et al (2020) to express general associative memory models using neural network dynamics with only second-order interactions between neurons, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.

Related papers

• Storing overlapping associative memories on latent manifolds in low-rank spiking networks [5.041384008847852]
  We revisit the associative memory problem in light of advances in understanding spike-based computation. We show that the spiking activity for a large class of all-inhibitory networks is situated on a low-dimensional, convex, and piecewise-linear manifold. We propose several learning rules, and demonstrate a linear scaling of the storage capacity with the number of neurons, as well as robust pattern completion abilities.
  arXiv  Detail & Related papers  (2024-11-26T14:48:25Z)
• Dense Associative Memory Through the Lens of Random Features [48.17520168244209]
  Dense Associative Memories are high storage capacity variants of the Hopfield networks. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories.
  arXiv  Detail & Related papers  (2024-10-31T17:10:57Z)
• B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory [91.81390121042192]
  We develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an composable module. B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens.
  arXiv  Detail & Related papers  (2024-07-08T18:41:01Z)
• Bridging Associative Memory and Probabilistic Modeling [29.605203018237457]
  Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. We build a bridge between the two that enables useful flow of ideas in both directions.
  arXiv  Detail & Related papers  (2024-02-15T18:56:46Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• In search of dispersed memories: Generative diffusion models are associative memory networks [6.4322891559626125]
  Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. We show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network.
  arXiv  Detail & Related papers  (2023-09-29T14:48:24Z)
• On the Relationship Between Variational Inference and Auto-Associative Memory [68.8204255655161]
  We study how different neural network approaches to variational inference can be applied in this framework. We evaluate the obtained algorithms on the CIFAR10 and CLEVR image datasets and compare them with other associative memory models.
  arXiv  Detail & Related papers  (2022-10-14T14:18:47Z)
• Kernel Memory Networks: A Unifying Framework for Memory Modeling [9.142894972380216]
  We consider the problem of training a neural network to store a set of patterns with maximal noise robustness. A solution is derived by training each individual neuron to perform either kernel classification or with a minimum weight norm. We derive optimal models, termed kernel memory networks, that include, as special cases, many of the hetero- and auto-associative memory models.
  arXiv  Detail & Related papers  (2022-08-19T16:01:09Z)
• Memory-Guided Semantic Learning Network for Temporal Sentence Grounding [55.31041933103645]
  We propose a memory-augmented network that learns and memorizes the rarely appeared content in TSG tasks. MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module.
  arXiv  Detail & Related papers  (2022-01-03T02:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.