Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture

• URL: http://arxiv.org/abs/2310.03052v3
• Date: Sat, 8 Jun 2024 04:17:55 GMT
• Title: Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture
• Authors: Sangjun Park, JinYeong Bak,
• Abstract summary: We present Memoria, a memory system for artificial neural networks. Results prove the effectiveness of Memoria in the diverse tasks of sorting, language modeling, and classification. Engram analysis reveals that Memoria exhibits the primacy, recency, and temporal contiguity effects which are characteristics of human memory.
• Score: 5.9360953869782325
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Making neural networks remember over the long term has been a longstanding issue. Although several external memory techniques have been introduced, most focus on retaining recent information in the short term. Regardless of its importance, information tends to be fatefully forgotten over time. We present Memoria, a memory system for artificial neural networks, drawing inspiration from humans and applying various neuroscientific and psychological theories. The experimental results prove the effectiveness of Memoria in the diverse tasks of sorting, language modeling, and classification, surpassing conventional techniques. Engram analysis reveals that Memoria exhibits the primacy, recency, and temporal contiguity effects which are characteristics of human memory.

Related papers

• Hierarchical Working Memory and a New Magic Number [1.024113475677323]
  We propose a recurrent neural network model for chunking within the framework of the synaptic theory of working memory. Our work provides a novel conceptual and analytical framework for understanding the on-the-fly organization of information in the brain that is crucial for cognition.
  arXiv  Detail & Related papers  (2024-08-14T16:03:47Z)
• Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models [49.39276272693035]
  Large-scale pre-trained language models have shown remarkable memorizing ability. Vanilla neural networks without pre-training have been long observed suffering from the catastrophic forgetting problem. We find that 1) Vanilla language models are forgetful; 2) Pre-training leads to retentive language models; 3) Knowledge relevance and diversification significantly influence the memory formation.
  arXiv  Detail & Related papers  (2023-05-16T03:50:38Z)
• Deep learning reveals the common spectrum underlying multiple brain disorders in youth and elders from brain functional networks [53.257804915263165]
  Brain disorders in the early and late life of humans potentially share pathological alterations in brain functions. Key evidence from neuroimaging data for pathological commonness remains unrevealed. We build a deep learning model, using multi-site functional magnetic resonance imaging data, for classifying 5 different brain disorders from healthy controls.
  arXiv  Detail & Related papers  (2023-02-23T09:22:05Z)
• Memory-Augmented Theory of Mind Network [59.9781556714202]
  Social reasoning requires the capacity of theory of mind (ToM) to contextualise and attribute mental states to others. Recent machine learning approaches to ToM have demonstrated that we can train the observer to read the past and present behaviours of other agents. We tackle the challenges by equipping the observer with novel neural memory mechanisms to encode, and hierarchical attention to selectively retrieve information about others. This results in ToMMY, a theory of mind model that learns to reason while making little assumptions about the underlying mental processes.
  arXiv  Detail & Related papers  (2023-01-17T14:48:58Z)
• Saliency-Augmented Memory Completion for Continual Learning [8.243137410556495]
  How to forget is a problem continual learning must address. Our paper proposes a new saliency-augmented memory completion framework for continual learning.
  arXiv  Detail & Related papers  (2022-12-26T18:06:39Z)
• A bio-inspired implementation of a sparse-learning spike-based hippocampus memory model [0.0]
  We propose a novel bio-inspired memory model based on the hippocampus. It can learn memories, recall them from a cue and even forget memories when trying to learn others with the same cue. This work presents the first hardware implementation of a fully functional bio-inspired spike-based hippocampus memory model.
  arXiv  Detail & Related papers  (2022-06-10T07:48:29Z)
• Latent Space based Memory Replay for Continual Learning in Artificial Neural Networks [0.0]
  We explore the application of latent space based memory replay for classification using artificial neural networks. We are able to preserve good performance in previous tasks by storing only a small percentage of the original data in a compressed latent space version.
  arXiv  Detail & Related papers  (2021-11-26T02:47:51Z)
• Memory and attention in deep learning [19.70919701635945]
  Memory construction for machine is inevitable. Recent progresses on modeling memory in deep learning have revolved around external memory constructions. The aim of this thesis is to advance the understanding on memory and attention in deep learning.
  arXiv  Detail & Related papers  (2021-07-03T09:21:13Z)
• Learning to Rehearse in Long Sequence Memorization [107.14601197043308]
  Existing reasoning tasks often have an important assumption that the input contents can be always accessed while reasoning. Memory augmented neural networks introduce a human-like write-read memory to compress and memorize the long input sequence in one pass. But they have two serious drawbacks: 1) they continually update the memory from current information and inevitably forget the early contents; 2) they do not distinguish what information is important and treat all contents equally. We propose the Rehearsal Memory to enhance long-sequence memorization by self-supervised rehearsal with a history sampler.
  arXiv  Detail & Related papers  (2021-06-02T11:58:30Z)
• Not All Memories are Created Equal: Learning to Forget by Expiring [49.053569908417636]
  We propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant information. This forgetting of memories enables Transformers to scale to attend over tens of thousands of previous timesteps efficiently. We show that Expire-Span can scale to memories that are tens of thousands in size, setting a new state of the art on incredibly long context tasks.
  arXiv  Detail & Related papers  (2021-05-13T20:50:13Z)
• Self-Attentive Associative Memory [69.40038844695917]
  We propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory) We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks.
  arXiv  Detail & Related papers  (2020-02-10T03:27:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.