Redundancy Maximization as a Principle of Associative Memory Learning

• URL: http://arxiv.org/abs/2511.02584v1
• Date: Tue, 04 Nov 2025 14:01:36 GMT
• Title: Redundancy Maximization as a Principle of Associative Memory Learning
• Authors: Mark Blümel, Andreas C. Schneider, Valentin Neuhaus, David A. Ehrlich, Marcel Graetz, Michael Wibral, Abdullah Makkeh, Viola Priesemann,
• Abstract summary: Associative memory, traditionally modeled by Hopfield networks, enables the retrieval of previously stored patterns.<n>We employ a recent extension of information theory - Partial Decomposition (PID)<n>We find that below the memory capacity, the information in a neuron's activity is characterized by high redundancy.
• Score: 2.0950316641796225
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Associative memory, traditionally modeled by Hopfield networks, enables the retrieval of previously stored patterns from partial or noisy cues. Yet, the local computational principles which are required to enable this function remain incompletely understood. To formally characterize the local information processing in such systems, we employ a recent extension of information theory - Partial Information Decomposition (PID). PID decomposes the contribution of different inputs to an output into unique information from each input, redundant information across inputs, and synergistic information that emerges from combining different inputs. Applying this framework to individual neurons in classical Hopfield networks we find that below the memory capacity, the information in a neuron's activity is characterized by high redundancy between the external pattern input and the internal recurrent input, while synergy and unique information are close to zero until the memory capacity is surpassed and performance drops steeply. Inspired by this observation, we use redundancy as an information-theoretic learning goal, which is directly optimized for each neuron, dramatically increasing the network's memory capacity to 1.59, a more than tenfold improvement over the 0.14 capacity of classical Hopfield networks and even outperforming recent state-of-the-art implementations of Hopfield networks. Ultimately, this work establishes redundancy maximization as a new design principle for associative memories and opens pathways for new associative memory models based on information-theoretic goals.

Related papers

• A Neural Network Model of Complementary Learning Systems: Pattern Separation and Completion for Continual Learning [2.9123921488295768]
  Learning new information without forgetting prior knowledge is central to human intelligence.<n>In contrast, neural network models suffer from catastrophic forgetting when acquiring new information.<n>We develop a neurally plausible continual learning model that achieves close to state-of-the-art accuracy (90%)<n>Our work provides a functional template for modeling memory consolidation, generalization, and continual learning in both biological and artificial systems.
  arXiv  Detail & Related papers  (2025-07-15T15:05:26Z)
• Latent Structured Hopfield Network for Semantic Association and Retrieval [52.634915010996835]
  Episodic memory enables humans to recall past experiences by associating semantic elements such as objects, locations, and time into coherent event representations.<n>We propose the Latent Structured Hopfield Network (LSHN), a framework that integrates continuous Hopfield attractor dynamics into an autoencoder architecture.<n>Unlike traditional Hopfield networks, our model is trained end-to-end with gradient descent, achieving scalable and robust memory retrieval.
  arXiv  Detail & Related papers  (2025-06-02T04:24:36Z)
• Modern Hopfield Networks with Continuous-Time Memories [19.616624959353697]
  We propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories.<n>Inspired by psychological theories of continuous neural resource allocation in working memory, we propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories.
  arXiv  Detail & Related papers  (2025-02-14T12:41:05Z)
• Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks [3.961279440272764]
  The Hopfield model provides a mathematically idealized yet insightful framework for understanding the mechanisms of memory storage and retrieval in the human brain. We propose a novel system framework in which the external input directly influences the neural synapses and shapes the energy landscape of the Hopfield model. This plasticity-based mechanism provides a clear energetic interpretation of the memory retrieval process and proves effective at correctly classifying highly mixed inputs.
  arXiv  Detail & Related papers  (2024-11-06T17:24:25Z)
• Dense Associative Memory Through the Lens of Random Features [48.17520168244209]
  Dense Associative Memories are high storage capacity variants of the Hopfield networks. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories.
  arXiv  Detail & Related papers  (2024-10-31T17:10:57Z)
• In search of dispersed memories: Generative diffusion models are associative memory networks [6.4322891559626125]
  Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. We show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network.
  arXiv  Detail & Related papers  (2023-09-29T14:48:24Z)
• Cross-Frequency Coupling Increases Memory Capacity in Oscillatory Neural Networks [69.42260428921436]
  Cross-frequency coupling (CFC) is associated with information integration across populations of neurons. We construct a model of CFC which predicts a computational role for observed $theta - gamma$ oscillatory circuits in the hippocampus and cortex. We show that the presence of CFC increases the memory capacity of a population of neurons connected by plastic synapses.
  arXiv  Detail & Related papers  (2022-04-05T17:13:36Z)
• HiPPO: Recurrent Memory with Optimal Polynomial Projections [93.3537706398653]
  We introduce a general framework (HiPPO) for the online compression of continuous signals and discrete time series by projection onto bases. Given a measure that specifies the importance of each time step in the past, HiPPO produces an optimal solution to a natural online function approximation problem. This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale.
  arXiv  Detail & Related papers  (2020-08-17T23:39:33Z)
• Distributed Associative Memory Network with Memory Refreshing Loss [5.5792083698526405]
  We introduce a novel Distributed Associative Memory architecture (DAM) with Memory Refreshing Loss (MRL) Inspired by how the human brain works, our framework encodes data with distributed representation across multiple memory blocks. MRL enables MANN to reinforce an association between input data and task objective by reproducing input data from stored memory contents.
  arXiv  Detail & Related papers  (2020-07-21T07:34:33Z)
• Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory [79.42778415729475]
  We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. We show how to extend the architecture of a simple RNN by separating its hidden state into different modules. We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
  arXiv  Detail & Related papers  (2020-06-29T08:35:49Z)
• Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations [143.3053365553897]
  We describe a procedure for removing dependency on a cohort of training data from a trained deep network. We introduce a new bound on how much information can be extracted per query about the forgotten cohort. We exploit the connections between the activation and weight dynamics of a DNN inspired by Neural Tangent Kernels to compute the information in the activations.
  arXiv  Detail & Related papers  (2020-03-05T23:17:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.