Dense Associative Memory with Epanechnikov Energy

• URL: http://arxiv.org/abs/2506.10801v1
• Date: Thu, 12 Jun 2025 15:16:31 GMT
• Title: Dense Associative Memory with Epanechnikov Energy
• Abstract summary: We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR)<n>Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel.<n> Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models.
• Score: 31.882557303410483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Moreover, it introduces abundant additional \emph{emergent} local minima while preserving perfect pattern recovery -- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks.

Related papers

• Free energy landscape of Dense Associative Memory [2.636047873931302]
  We solve and obtain a general expression for the free energy functional for a broad class of associative memories.<n>For a finite number of patterns, we derive the temperature-dependent free energy functional for dense associative memories.<n>Our framework reveals how memory retrieval depends on the initial state in higher-order dense networks.
  arXiv  Detail & Related papers  (2026-07-21T15:30:53Z)
• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents [51.30250860677378]
  We propose MRAgent, a framework that combines an associative memory graph with an active reconstruction mechanism.<n>Running on a Cue-Tag-Content graph, our active reconstruction mechanism integrates reasoning directly into memory access.<n>Experiments on the LoCoMo benchmark and LongMemEval benchmark demonstrate significant improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T11:29:46Z)
• Neural Dynamics Self-Attention for Spiking Transformers [59.57278823240336]
  Spiking Neural Networks (SNNs) with Transformer architectures offer a promising pathway to balance energy efficiency and performance.<n>Existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference.<n>We propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements.
  arXiv  Detail & Related papers  (2026-03-09T12:23:37Z)
• MemRec: Collaborative Memory-Augmented Agentic Recommender System [57.548438733740504]
  We propose MemRec, a framework that architecturally decouples reasoning from memory management.<n>MemRec introduces a dedicated LM_Mem to manage a dynamic collaborative memory graph.<n>It achieves state-of-the-art performance on four benchmarks.
  arXiv  Detail & Related papers  (2026-01-13T18:51:16Z)
• Improving Multi-step RAG with Hypergraph-based Memory for Long-Context Complex Relational Modeling [83.29209853451697]
  Multi-step retrieval-augmented generation (RAG) has become a widely adopted strategy for enhancing large language models (LLMs)<n>We introduce HGMem, a hypergraph-based memory mechanism that extends the concept of memory into a dynamic, expressive structure for complex reasoning and global understanding.<n>In our approach, memory is represented as a hypergraph whose hyperedges correspond to distinct memory units, enabling the progressive formation of higher-order interactions within memory.
  arXiv  Detail & Related papers  (2025-12-30T03:13:10Z)
• Exponential Dynamic Energy Network for High Capacity Sequence Memory [11.724565818034948]
  We introduce the Exponential Dynamic Energy Network (EDEN), a novel architecture that extends the energy paradigm to temporal domains.<n>EDEN offers a scalable and interpretable model for high-capacity temporal memory in both artificial and biological systems.
  arXiv  Detail & Related papers  (2025-10-28T20:53:24Z)
• Effects of Feature Correlations on Associative Memory Capacity [1.024113475677323]
  We develop an empirical framework to analyze the effects of data structure on capacity dynamics.<n>Experiments confirm that memory capacity scales exponentially with increasing separation in the input space.<n>Our findings bridge theoretical work and practical settings for DAM, and might inspire more data-centric methods.
  arXiv  Detail & Related papers  (2025-08-02T15:03:01Z)
• Higher-Order Kuramoto Oscillator Network for Dense Associative Memory [0.0]
  We show that higher-order couplings achieve superlinear scaling of memory capacity with system size.<n>These results bridge the Kuramoto synchronization with modern Hopfield memories.
  arXiv  Detail & Related papers  (2025-07-29T16:35:52Z)
• On the role of non-linear latent features in bipartite generative neural networks [4.499833362998488]
  We investigate Restricted Boltzmann Machines (RBMs) as a function of the prior distribution imposed on their hidden units.<n>Our analysis reveals that standard RBMs with binary hidden nodes and extensive connectivity suffer from reduced critical capacity.<n>To address this, we examine several modifications, such as introducing local biases and adopting richer hidden unit priors.
  arXiv  Detail & Related papers  (2025-06-12T10:20:20Z)
• Joint Learning of Energy-based Models and their Partition Function [19.174145933837927]
  Energy-based models (EBMs) offer a flexible framework for parameterizing probability distributions using neural networks.<n>We propose a novel formulation for approximately learning EBMs inly-large discrete spaces.<n>We show that our approach naturally extends to the broader family of Fenchel-Young losses.
  arXiv  Detail & Related papers  (2025-01-30T17:46:17Z)
• CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation [63.65323577445951]
  We propose a novel approach called Cache Sparse Representation (CSR)<n>CSR transforms the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference.<n>Our experiments demonstrate CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms.
  arXiv  Detail & Related papers  (2024-12-16T13:01:53Z)
• Estimation of Energy-dissipation Lower-bounds for Neuromorphic Learning-in-memory [5.073292775065559]
  An ideal neuromorphic neurally-inspired neurally-equilibriums rely on local but parallel parameter updates to solve problems that range from quadratic programming to Ising machines.<n>An analysis presented in this paper captures the out-of- thermodynamics of learning and the resulting energy-efficiency estimates are model-agnostic.<n>To show practical applicability of our results, we apply our analysis for estimating the lower-bound on the energy-to-solution metrics for large-scale AI workloads.
  arXiv  Detail & Related papers  (2024-02-21T21:02:11Z)
• Bridging Associative Memory and Probabilistic Modeling [29.605203018237457]
  Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. We build a bridge between the two that enables useful flow of ideas in both directions.
  arXiv  Detail & Related papers  (2024-02-15T18:56:46Z)
• Topology-aware Embedding Memory for Continual Learning on Expanding Networks [63.35819388164267]
  We present a framework to tackle the memory explosion problem using memory replay techniques. PDGNNs with Topology-aware Embedding Memory (TEM) significantly outperform state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-01-24T03:03:17Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Selective Memory Recursive Least Squares: Recast Forgetting into Memory in RBF Neural Network Based Real-Time Learning [2.31120983784623]
  In radial basis function neural network (RBFNN) based real-time learning tasks, forgetting mechanisms are widely used. This paper proposes a real-time training method named selective memory recursive least squares (SMRLS) in which the classical forgetting mechanisms are recast into a memory mechanism. With SMRLS, the input space of the RBFNN is evenly divided into a finite number of partitions and a synthesized objective function is developed using synthesized samples from each partition.
  arXiv  Detail & Related papers  (2022-11-15T05:29:58Z)
• Learning Optical Flow from a Few Matches [67.83633948984954]
  We show that the dense correlation volume representation is redundant and accurate flow estimation can be achieved with only a fraction of elements in it. Experiments show that our method can reduce computational cost and memory use significantly, while maintaining high accuracy.
  arXiv  Detail & Related papers  (2021-04-05T21:44:00Z)
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory [75.65949969000596]
  Episodic and semantic memory are critical components of the human memory model. We develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation.
  arXiv  Detail & Related papers  (2021-02-20T18:40:40Z)
• Deep Representational Similarity Learning for analyzing neural signatures in task-based fMRI dataset [81.02949933048332]
  This paper develops Deep Representational Similarity Learning (DRSL), a deep extension of Representational Similarity Analysis (RSA) DRSL is appropriate for analyzing similarities between various cognitive tasks in fMRI datasets with a large number of subjects.
  arXiv  Detail & Related papers  (2020-09-28T18:30:14Z)
• Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining [66.82470461139376]
  We propose the first Cross-Scale Non-Local (CS-NL) attention module with integration into a recurrent neural network. By combining the new CS-NL prior with local and in-scale non-local priors in a powerful recurrent fusion cell, we can find more cross-scale feature correlations within a single low-resolution image.
  arXiv  Detail & Related papers  (2020-06-02T07:08:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.