Dense Associative Memory Through the Lens of Random Features

• URL: http://arxiv.org/abs/2410.24153v1
• Date: Thu, 31 Oct 2024 17:10:57 GMT
• Title: Dense Associative Memory Through the Lens of Random Features
• Authors: Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, Parikshit Ram, Dmitry Krotov,
• Abstract summary: Dense Associative Memories are high storage capacity variants of the Hopfield networks. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories.
• Score: 48.17520168244209
• License:
• Abstract: Dense Associative Memories are high storage capacity variants of the Hopfield networks that are capable of storing a large number of memory patterns in the weights of the network of a given size. Their common formulations typically require storing each pattern in a separate set of synaptic weights, which leads to the increase of the number of synaptic weights when new patterns are introduced. In this work we propose an alternative formulation of this class of models using random features, commonly used in kernel methods. In this formulation the number of network's parameters remains fixed. At the same time, new memories can be added to the network by modifying existing weights. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories and shares their desirable computational properties.

Related papers

• Dr$^2$Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning [81.0108753452546]
  We propose Dynamic Reversible Dual-Residual Networks, or Dr$2$Net, to finetune a pretrained model with substantially reduced memory consumption. Dr$2$Net contains two types of residual connections, one maintaining the residual structure in the pretrained models, and the other making the network reversible. We show that Dr$2$Net can reach comparable performance to conventional finetuning but with significantly less memory usage.
  arXiv  Detail & Related papers  (2024-01-08T18:59:31Z)
• Long Sequence Hopfield Memory [32.28395813801847]
  Sequence memory enables agents to encode, store, and retrieve complex sequences of stimuli and actions. We introduce a nonlinear interaction term, enhancing separation between the patterns. We extend this model to store sequences with variable timing between states' transitions.
  arXiv  Detail & Related papers  (2023-06-07T15:41:03Z)
• Simplicial Hopfield networks [0.0]
  We extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. We show that our simplicial Hopfield networks increase memory storage capacity. We also test analogous modern continuous Hopfield networks, offering a potentially promising avenue for improving the attention mechanism in Transformer models.
  arXiv  Detail & Related papers  (2023-05-09T05:23:04Z)
• On the Relationship Between Variational Inference and Auto-Associative Memory [68.8204255655161]
  We study how different neural network approaches to variational inference can be applied in this framework. We evaluate the obtained algorithms on the CIFAR10 and CLEVR image datasets and compare them with other associative memory models.
  arXiv  Detail & Related papers  (2022-10-14T14:18:47Z)
• Parameter-Efficient Masking Networks [61.43995077575439]
  Advanced network designs often contain a large number of repetitive structures (e.g., Transformer) In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning masks. It leads to a new paradigm for model compression to diminish the model size.
  arXiv  Detail & Related papers  (2022-10-13T03:39:03Z)
• Kernel Memory Networks: A Unifying Framework for Memory Modeling [9.142894972380216]
  We consider the problem of training a neural network to store a set of patterns with maximal noise robustness. A solution is derived by training each individual neuron to perform either kernel classification or with a minimum weight norm. We derive optimal models, termed kernel memory networks, that include, as special cases, many of the hetero- and auto-associative memory models.
  arXiv  Detail & Related papers  (2022-08-19T16:01:09Z)
• Neural Computing with Coherent Laser Networks [0.0]
  We show that a coherent network of lasers exhibits emergent neural computing capabilities. A novel energy-based recurrent neural network handles continuous data as opposed to Hopfield networks and Boltzmann machines.
  arXiv  Detail & Related papers  (2022-04-05T13:56:34Z)
• Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models [41.58529335439799]
  We propose a general framework for understanding the operation of memory networks as a sequence of three operations. We derive all these memory models as instances of our general framework with differing similarity and separation functions.
  arXiv  Detail & Related papers  (2022-02-09T16:48:06Z)
• Neural Network Compression for Noisy Storage Devices [71.4102472611862]
  Conventionally, model compression and physical storage are decoupled. This approach forces the storage to treat each bit of the compressed model equally, and to dedicate the same amount of resources to each bit. We propose a radically different approach that: (i) employs analog memories to maximize the capacity of each memory cell, and (ii) jointly optimize model compression and physical storage to maximize memory utility.
  arXiv  Detail & Related papers  (2021-02-15T18:19:07Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.