Predictive Attractor Models

• URL: http://arxiv.org/abs/2410.02430v1
• Date: Thu, 3 Oct 2024 12:25:01 GMT
• Title: Predictive Attractor Models
• Authors: Ramy Mounir, Sudeep Sarkar,
• Abstract summary: We propose textitPredictive Attractor Models (PAM), a novel sequence memory architecture with desirable generative properties. PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework.
• Score: 9.947717243638289
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sequential memory, the ability to form and accurately recall a sequence of events or stimuli in the correct order, is a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language comprehension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose \textit{Predictive Attractor Models (PAM)}, a novel sequence memory architecture with desirable generative properties. PAM is a streaming model that learns a sequence in an online, continuous manner by observing each input \textit{only once}. Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from a union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g., noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PAM represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intelligence research.

Related papers

• Causal Estimation of Memorisation Profiles [58.20086589761273]
  Understanding memorisation in language models has practical and societal implications. Memorisation is the causal effect of training with an instance on the model's ability to predict that instance. This paper proposes a new, principled, and efficient method to estimate memorisation based on the difference-in-differences design from econometrics.
  arXiv  Detail & Related papers  (2024-06-06T17:59:09Z)
• Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling [53.7117640028211]
  We present a geometry-constrained probabilistic modeling treatment to resolve the identified issues. We incorporate a suite of critical geometric properties to impose proper constraints on the layout of constructed embedding space. A spectral graph-theoretic method is devised to estimate the number of potential novel classes.
  arXiv  Detail & Related papers  (2024-03-02T00:56:05Z)
• Memory, Space, and Planning: Multiscale Predictive Representations [5.572701755354684]
  Flexible behavior in biological and artificial agents depends on the interplay of learning from the past and predicting the future in ever-changing environments. This chapter reviews computational, behavioral, and neural evidence suggesting these processes rely on learning the structure of experiences, known as cognitive maps.
  arXiv  Detail & Related papers  (2024-01-16T21:46:43Z)
• Memory-and-Anticipation Transformer for Online Action Understanding [52.24561192781971]
  We propose a novel memory-anticipation-based paradigm to model an entire temporal structure, including the past, present, and future. We present Memory-and-Anticipation Transformer (MAT), a memory-anticipation-based approach, to address the online action detection and anticipation tasks.
  arXiv  Detail & Related papers  (2023-08-15T17:34:54Z)
• Computation with Sequences in a Model of the Brain [11.15191997898358]
  How cognition arises from neural activity is a central open question in neuroscience. We show that time can be captured naturally as precedence through synaptic weights and plasticity. We show that any finite state machine can be learned in a similar way, through the presentation of appropriate patterns of sequences.
  arXiv  Detail & Related papers  (2023-06-06T15:58:09Z)
• Sequential Memory with Temporal Predictive Coding [6.228559238589584]
  We propose a PC-based model for emphsequential memory, called emphtemporal predictive coding (tPC) We show that our tPC models can memorize and retrieve sequential inputs accurately with a biologically plausible neural implementation.
  arXiv  Detail & Related papers  (2023-05-19T20:03:31Z)
• Sequence Learning and Consolidation on Loihi using On-chip Plasticity [6.9597705368779925]
  We develop a model of predictive learning on neuromorphic hardware using the on-chip plasticity capabilities of the Loihi chip. Our model serves as a proof-of-concept that online predictive learning models can be deployed on neuromorphic hardware with on-chip plasticity.
  arXiv  Detail & Related papers  (2022-05-02T04:18:50Z)
• Learning Theory of Mind via Dynamic Traits Attribution [59.9781556714202]
  We propose a new neural ToM architecture that learns to generate a latent trait vector of an actor from the past trajectories. This trait vector then multiplicatively modulates the prediction mechanism via a fast weights' scheme in the prediction neural network. We empirically show that the fast weights provide a good inductive bias to model the character traits of agents and hence improves mindreading ability.
  arXiv  Detail & Related papers  (2022-04-17T11:21:18Z)
• CogNGen: Constructing the Kernel of a Hyperdimensional Predictive Processing Cognitive Architecture [79.07468367923619]
  We present a new cognitive architecture that combines two neurobiologically plausible, computational models. We aim to develop a cognitive architecture that has the power of modern machine learning techniques.
  arXiv  Detail & Related papers  (2022-03-31T04:44:28Z)
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution [97.50813120600026]
  Spatial-temporal reasoning is a challenging task in Artificial Intelligence (AI) Recent works have focused on an abstract reasoning task of this kind -- Raven's Progressive Matrices ( RPM) We propose a neuro-symbolic Probabilistic Abduction and Execution learner (PrAE) learner.
  arXiv  Detail & Related papers  (2021-03-26T02:42:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.