Prototype Analysis in Hopfield Networks with Hebbian Learning

• URL: http://arxiv.org/abs/2407.03342v1
• Date: Wed, 29 May 2024 01:03:48 GMT
• Title: Prototype Analysis in Hopfield Networks with Hebbian Learning
• Authors: Hayden McAlister, Anthony Robins, Lech Szymanski,
• Abstract summary: Hebbian learning with highly correlated states leads to degraded memory performance. We show this type of learning can lead to prototype formation, where unlearned states emerge as representatives of large correlated subsets of states. This process has similarities to prototype learning in human cognition.
• Score: 1.2289361708127877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We discuss prototype formation in the Hopfield network. Typically, Hebbian learning with highly correlated states leads to degraded memory performance. We show this type of learning can lead to prototype formation, where unlearned states emerge as representatives of large correlated subsets of states, alleviating capacity woes. This process has similarities to prototype learning in human cognition. We provide a substantial literature review of prototype learning in associative memories, covering contributions from psychology, statistical physics, and computer science. We analyze prototype formation from a theoretical perspective and derive a stability condition for these states based on the number of examples of the prototype presented for learning, the noise in those examples, and the number of non-example states presented. The stability condition is used to construct a probability of stability for a prototype state as the factors of stability change. We also note similarities to traditional network analysis, allowing us to find a prototype capacity. We corroborate these expectations of prototype formation with experiments using a simple Hopfield network with standard Hebbian learning. We extend our experiments to a Hopfield network trained on data with multiple prototypes and find the network is capable of stabilizing multiple prototypes concurrently. We measure the basins of attraction of the multiple prototype states, finding attractor strength grows with the number of examples and the agreement of examples. We link the stability and dominance of prototype states to the energy profile of these states, particularly when comparing the profile shape to target states or other spurious states.

Related papers

• Classifying States of the Hopfield Network with Improved Accuracy, Generalization, and Interpretability [1.2289361708127877]
  We study the generalizability of different classification models when trained on states derived from different prototype tasks. We find that simple models often outperform the stability ratio while remaining interpretable.
  arXiv  Detail & Related papers  (2025-03-04T21:29:42Z)
• Generative Modeling with Bayesian Sample Inference [50.07758840675341]
  We derive a novel generative model from the simple act of Gaussian posterior inference. Treating the generated sample as an unknown variable to infer lets us formulate the sampling process in the language of Bayesian probability. Our model uses a sequence of prediction and posterior update steps to narrow down the unknown sample from a broad initial belief.
  arXiv  Detail & Related papers  (2025-02-11T14:27:10Z)
• Sparse Prototype Network for Explainable Pedestrian Behavior Prediction [60.80524827122901]
  We present Sparse Prototype Network (SPN), an explainable method designed to simultaneously predict a pedestrian's future action, trajectory, and pose. Regularized by mono-semanticity and clustering constraints, the prototypes learn consistent and human-understandable features.
  arXiv  Detail & Related papers  (2024-10-16T03:33:40Z)
• Predefined Prototypes for Intra-Class Separation and Disentanglement [10.005120138175206]
  Prototypical Learning is based on the idea that there is a point (which we call prototype) around which the embeddings of a class are clustered. We propose to predefine prototypes following human-specified criteria, which simplify the training pipeline and brings different advantages.
  arXiv  Detail & Related papers  (2024-06-23T15:52:23Z)
• Interpreting What Typical Fault Signals Look Like via Prototype-matching [3.774984871230879]
  Prototype matching network (PMN) is proposed by combining the human-inherent prototype-matching with autoencoder (AE) It has three interpreting paths on classification logic, fault prototypes, and matching contributions. This ability broadens human understanding and provides a promising solution from interpretability research to AI-for-Science.
  arXiv  Detail & Related papers  (2024-03-11T05:47:07Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Learning to Select Prototypical Parts for Interpretable Sequential Data Modeling [7.376829794171344]
  We propose a Self-Explaining Selective Model (SESM) that uses a linear combination of prototypical concepts to explain its own predictions. For better interpretability, we design multiple constraints including diversity, stability, and locality as training objectives.
  arXiv  Detail & Related papers  (2022-12-07T01:42:47Z)
• Towards Human-Interpretable Prototypes for Visual Assessment of Image Classification Models [9.577509224534323]
  We need models which are interpretable-by-design built on a reasoning process similar to humans. ProtoPNet claims to discover visually meaningful prototypes in an unsupervised way. We find that these prototypes still have a long way ahead towards definite explanations.
  arXiv  Detail & Related papers  (2022-11-22T11:01:22Z)
• TTAPS: Test-Time Adaption by Aligning Prototypes using Self-Supervision [70.05605071885914]
  We propose a novel modification of the self-supervised training algorithm SwAV that adds the ability to adapt to single test samples. We show the success of our method on the common benchmark dataset CIFAR10-C.
  arXiv  Detail & Related papers  (2022-05-18T05:43:06Z)
• Interpretable Image Classification with Differentiable Prototypes Assignment [7.660883761395447]
  We introduce ProtoPool, an interpretable image classification model with a pool of prototypes shared by the classes. It is obtained by introducing a fully differentiable assignment of prototypes to particular classes. We show that ProtoPool obtains state-of-the-art accuracy on the CUB-200-2011 and the Stanford Cars datasets, substantially reducing the number of prototypes.
  arXiv  Detail & Related papers  (2021-12-06T10:03:32Z)
• On the Transferability of Adversarial Attacksagainst Neural Text Classifier [121.6758865857686]
  We investigate the transferability of adversarial examples for text classification models. We propose a genetic algorithm to find an ensemble of models that can induce adversarial examples to fool almost all existing models. We derive word replacement rules that can be used for model diagnostics from these adversarial examples.
  arXiv  Detail & Related papers  (2020-11-17T10:45:05Z)
• Toward Scalable and Unified Example-based Explanation and Outlier Detection [128.23117182137418]
  We argue for a broader adoption of prototype-based student networks capable of providing an example-based explanation for their prediction. We show that our prototype-based networks beyond similarity kernels deliver meaningful explanations and promising outlier detection results without compromising classification accuracy.
  arXiv  Detail & Related papers  (2020-11-11T05:58:17Z)
• Learning Sparse Prototypes for Text Generation [120.38555855991562]
  Prototype-driven text generation is inefficient at test time as a result of needing to store and index the entire training corpus. We propose a novel generative model that automatically learns a sparse prototype support set that achieves strong language modeling performance. In experiments, our model outperforms previous prototype-driven language models while achieving up to a 1000x memory reduction.
  arXiv  Detail & Related papers  (2020-06-29T19:41:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.