Effects of Feature Correlations on Associative Memory Capacity

• URL: http://arxiv.org/abs/2508.01395v1
• Date: Sat, 02 Aug 2025 15:03:01 GMT
• Title: Effects of Feature Correlations on Associative Memory Capacity
• Authors: Stefan Bielmeier, Gerald Friedland,
• Abstract summary: We develop an empirical framework to analyze the effects of data structure on capacity dynamics.<n>Experiments confirm that memory capacity scales exponentially with increasing separation in the input space.<n>Our findings bridge theoretical work and practical settings for DAM, and might inspire more data-centric methods.
• Score: 1.024113475677323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate how feature correlations influence the capacity of Dense Associative Memory (DAM), a Transformer attention-like model. Practical machine learning scenarios involve feature-correlated data and learn representations in the input space, but current capacity analyses do not account for this. We develop an empirical framework to analyze the effects of data structure on capacity dynamics. Specifically, we systematically construct datasets that vary in feature correlation and pattern separation using Hamming distance from information theory, and compute the model's corresponding storage capacity using a simple binary search algorithm. Our experiments confirm that memory capacity scales exponentially with increasing separation in the input space. Feature correlations do not alter this relationship fundamentally, but reduce capacity slightly at constant separation. This effect is amplified at higher polynomial degrees in the energy function, suggesting that Associative Memory is more limited in depicting higher-order interactions between features than patterns. Our findings bridge theoretical work and practical settings for DAM, and might inspire more data-centric methods.

Related papers

• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data [6.885357232728911]
  This paper studies how the model architecture and data configurations influence the empirical memorization capacity of generative transformers.<n>The models are trained using synthetic text datasets derived from the Systematized Nomenclature of Medicine (SNOMED)
  arXiv  Detail & Related papers  (2025-06-17T16:42:54Z)
• A Computational Model of Learning and Memory Using Structurally Dynamic Cellular Automata [0.0]
  This paper proposes a mathematical and computational model of learning and memory based on a small set of bio-plausible functions.<n> Experimental results show that the model can make near-optimal choices to re-discover a reward state after a single training run.
  arXiv  Detail & Related papers  (2024-12-20T17:26:17Z)
• Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions [0.552480439325792]
  storage capacity of a binary classification model is the maximum number of random input-output pairs per parameter that the model can learn.<n>We analyze the structure of the solution space and the storage capacity of fully connected two-layer neural networks with general activation functions.
  arXiv  Detail & Related papers  (2024-04-20T15:12:47Z)
• Learning Associative Memories with Gradient Descent [21.182801606213495]
  This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings. We show that imbalance in token frequencies and memory interferences due to correlated embeddings lead to transitory regimes.
  arXiv  Detail & Related papers  (2024-02-28T21:47:30Z)
• Bridging Associative Memory and Probabilistic Modeling [29.605203018237457]
  Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. We build a bridge between the two that enables useful flow of ideas in both directions.
  arXiv  Detail & Related papers  (2024-02-15T18:56:46Z)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression [53.15502562048627]
  Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator. This work delves into a statistical analysis of augmentation-based pretraining.
  arXiv  Detail & Related papers  (2023-06-01T15:18:55Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Towards Differential Relational Privacy and its use in Question Answering [109.4452196071872]
  Memorization of relation between entities in a dataset can lead to privacy issues when using a trained question answering model. We quantify this phenomenon and provide a possible definition of Differential Privacy (DPRP) We illustrate concepts in experiments with largescale models for Question Answering.
  arXiv  Detail & Related papers  (2022-03-30T22:59:24Z)
• Learning Optical Flow from a Few Matches [67.83633948984954]
  We show that the dense correlation volume representation is redundant and accurate flow estimation can be achieved with only a fraction of elements in it. Experiments show that our method can reduce computational cost and memory use significantly, while maintaining high accuracy.
  arXiv  Detail & Related papers  (2021-04-05T21:44:00Z)
• Optimal Learning with Excitatory and Inhibitory synapses [91.3755431537592]
  I study the problem of storing associations between analog signals in the presence of correlations. I characterize the typical learning performance in terms of the power spectrum of random input and output processes.
  arXiv  Detail & Related papers  (2020-05-25T18:25:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.