Modern Hopfield Networks with Continuous-Time Memories

• URL: http://arxiv.org/abs/2502.10122v1
• Date: Fri, 14 Feb 2025 12:41:05 GMT
• Title: Modern Hopfield Networks with Continuous-Time Memories
• Authors: Saul Santos, António Farinhas, Daniel C. McNamee, André F. T. Martins,
• Abstract summary: We propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories. Inspired by psychological theories of continuous neural resource allocation in working memory, we propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories.
• Score: 19.616624959353697
• License:
• Abstract: Recent research has established a connection between modern Hopfield networks (HNs) and transformer attention heads, with guarantees of exponential storage capacity. However, these models still face challenges scaling storage efficiently. Inspired by psychological theories of continuous neural resource allocation in working memory, we propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories. Leveraging continuous attention, our new energy function modifies the update rule of HNs, replacing the traditional softmax-based probability mass function with a probability density, over the continuous memory. This formulation aligns with modern perspectives on human executive function, offering a principled link between attractor dynamics in working memory and resource-efficient memory allocation. Our framework maintains competitive performance with HNs while leveraging a compressed memory, reducing computational costs across synthetic and video datasets.

Related papers

• Exploring Synaptic Resonance in Large Language Models: A Novel Approach to Contextual Memory Integration [0.0]
  A novel mechanism, Synaptic Resonance, is introduced to dynamically reinforce relevant memory pathways during training and inference. Evaluations conducted on an open-source language model demonstrate reductions in perplexity, enhancements in contextual coherence, and increased robustness against input noise.
  arXiv  Detail & Related papers  (2025-02-15T07:06:10Z)
• Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks [3.961279440272764]
  The Hopfield model provides a mathematically idealized yet insightful framework for understanding the mechanisms of memory storage and retrieval in the human brain. We propose a novel system framework in which the external input directly influences the neural synapses and shapes the energy landscape of the Hopfield model. This plasticity-based mechanism provides a clear energetic interpretation of the memory retrieval process and proves effective at correctly classifying highly mixed inputs.
  arXiv  Detail & Related papers  (2024-11-06T17:24:25Z)
• Dense Associative Memory Through the Lens of Random Features [48.17520168244209]
  Dense Associative Memories are high storage capacity variants of the Hopfield networks. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories.
  arXiv  Detail & Related papers  (2024-10-31T17:10:57Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)
• Resistive Memory-based Neural Differential Equation Solver for Score-based Diffusion Model [55.116403765330084]
  Current AIGC methods, such as score-based diffusion, are still deficient in terms of rapidity and efficiency. We propose a time-continuous and analog in-memory neural differential equation solver for score-based diffusion. We experimentally validate our solution with 180 nm resistive memory in-memory computing macros.
  arXiv  Detail & Related papers  (2024-04-08T16:34:35Z)
• Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models [5.929540708452128]
  We propose a two-stage memory retrieval dynamics for modern Hopfield models. Key contribution is a learnable feature map $Phi$ which transforms the Hopfield energy function into kernel space. It utilizes the stored memory patterns as learning data to enhance memory capacity across all modern Hopfield models.
  arXiv  Detail & Related papers  (2024-04-04T23:05:30Z)
• Constant Memory Attention Block [74.38724530521277]
  Constant Memory Attention Block (CMAB) is a novel general-purpose attention block that computes its output in constant memory and performs updates in constant computation. We show our proposed methods achieve results competitive with state-of-the-art while being significantly more memory efficient.
  arXiv  Detail & Related papers  (2023-06-21T22:41:58Z)
• Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models [41.58529335439799]
  We propose a general framework for understanding the operation of memory networks as a sequence of three operations. We derive all these memory models as instances of our general framework with differing similarity and separation functions.
  arXiv  Detail & Related papers  (2022-02-09T16:48:06Z)
• Temporal Memory Relation Network for Workflow Recognition from Surgical Video [53.20825496640025]
  We propose a novel end-to-end temporal memory relation network (TMNet) for relating long-range and multi-scale temporal patterns. We have extensively validated our approach on two benchmark surgical video datasets.
  arXiv  Detail & Related papers  (2021-03-30T13:20:26Z)
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory [75.65949969000596]
  Episodic and semantic memory are critical components of the human memory model. We develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation.
  arXiv  Detail & Related papers  (2021-02-20T18:40:40Z)
• Slow manifolds in recurrent networks encode working memory efficiently and robustly [0.0]
  Working memory is a cognitive function involving the storage and manipulation of latent information over brief intervals of time. We use a top-down modeling approach to examine network-level mechanisms of working memory.
  arXiv  Detail & Related papers  (2021-01-08T18:47:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.