Nonparametric Modern Hopfield Models

• URL: http://arxiv.org/abs/2404.03900v1
• Date: Fri, 5 Apr 2024 05:46:20 GMT
• Title: Nonparametric Modern Hopfield Models
• Authors: Jerry Yao-Chieh Hu, Bo-Yu Chen, Dennis Wu, Feng Ruan, Han Liu,
• Abstract summary: We present a nonparametric construction for deep learning compatible modern Hopfield models. Key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models. We introduce textitsparse-structured modern Hopfield models with sub-quadratic complexity.
• Score: 12.160725212848137
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a nonparametric construction for deep learning compatible modern Hopfield models and utilize this framework to debut an efficient variant. Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs. Crucially, our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient modern Hopfield models, by introducing \textit{sparse-structured} modern Hopfield models with sub-quadratic complexity. We establish that this sparse model inherits the appealing theoretical properties of its dense analogue -- connection with transformer attention, fixed point convergence and exponential memory capacity -- even without knowing details of the Hopfield energy function. Additionally, we showcase the versatility of our framework by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-$K$ and positive random feature modern Hopfield models. Empirically, we validate the efficacy of our framework in both synthetic and realistic settings.

Related papers

• Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval [25.841394444834933]
  Associative memory models, such as Hopfield networks, have garnered renewed interest due to advancements in memory capacity and connections with self-attention in transformers. In this work, we introduce a unified framework-Hopfield-Fenchel-Young networks-which generalizes these models to a broader family of energy functions.
  arXiv  Detail & Related papers  (2024-11-13T13:13:07Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Outlier-Efficient Hopfield Layers for Large Transformer-Based Models [10.972020273638066]
  We introduce an Outlier-Efficient Modern Hopfield Model (termed $mathrmOutEffHop$) Our main contribution is a novel associative memory model facilitating textitoutlier-efficient associative memory retrievals. $mathrmOutEffHop$ achieves an average reduction of 22+% in average kurtosis and 26+% in the maximum infinity norm of model outputs.
  arXiv  Detail & Related papers  (2024-04-04T23:08:43Z)
• On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis [12.72277128564391]
  We investigate the computational limits of the memory retrieval dynamics of modern Hopfield models. We establish an upper bound criterion for the norm of input query patterns and memory patterns. We prove its memory retrieval error bound and exponential memory capacity.
  arXiv  Detail & Related papers  (2024-02-07T01:58:21Z)
• STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction [13.815793371488613]
  We present a novel Hopfield-based neural network block, which sparsely learns and stores both temporal and cross-series representations. In essence, STanHop sequentially learn temporal representation and cross-series representation using two tandem sparse Hopfield layers. We show that our framework endows a tighter memory retrieval error compared to the dense counterpart without sacrificing memory capacity.
  arXiv  Detail & Related papers  (2023-12-28T20:26:23Z)
• On Sparse Modern Hopfield Model [12.288884253562845]
  We introduce the sparse modern Hopfield model as a sparse extension of the modern Hopfield model. We show that the sparse modern Hopfield model maintains the robust theoretical properties of its dense counterpart.
  arXiv  Detail & Related papers  (2023-09-22T07:32:45Z)
• Understanding Parameter Sharing in Transformers [53.75988363281843]
  Previous work on Transformers has focused on sharing parameters in different layers, which can improve the performance of models with limited parameters by increasing model depth. We show that the success of this approach can be largely attributed to better convergence, with only a small part due to the increased model complexity. Experiments on 8 machine translation tasks show that our model achieves competitive performance with only half the model complexity of parameter sharing models.
  arXiv  Detail & Related papers  (2023-06-15T10:48:59Z)
• Counting Phases and Faces Using Bayesian Thermodynamic Integration [77.34726150561087]
  We introduce a new approach to reconstruction of the thermodynamic functions and phase boundaries in two-parametric statistical mechanics systems. We use the proposed approach to accurately reconstruct the partition functions and phase diagrams of the Ising model and the exactly solvable non-equilibrium TASEP.
  arXiv  Detail & Related papers  (2022-05-18T17:11:23Z)
• Low-Rank Constraints for Fast Inference in Structured Models [110.38427965904266]
  This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models. Experiments with neural parameterized structured models for language modeling, polyphonic music modeling, unsupervised grammar induction, and video modeling show that our approach matches the accuracy of standard models at large state spaces.
  arXiv  Detail & Related papers  (2022-01-08T00:47:50Z)
• Mean Field Game GAN [55.445402222849474]
  We propose a novel mean field games (MFGs) based GAN(generative adversarial network) framework. We utilize the Hopf formula in density space to rewrite MFGs as a primal-dual problem so that we are able to train the model via neural networks and samples.
  arXiv  Detail & Related papers  (2021-03-14T06:34:38Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.