MEMO: A Deep Network for Flexible Combination of Episodic Memories
- URL: http://arxiv.org/abs/2001.10913v1
- Date: Wed, 29 Jan 2020 15:56:16 GMT
- Title: MEMO: A Deep Network for Flexible Combination of Episodic Memories
- Authors: Andrea Banino, Adri\`a Puigdom\`enech Badia, Raphael K\"oster, Martin
J. Chadwick, Vinicius Zambaldi, Demis Hassabis, Caswell Barry, Matthew
Botvinick, Dharshan Kumaran, Charles Blundell
- Abstract summary: MEMO is an architecture endowed with the capacity to reason over longer distances.
First, it introduces a separation between memories stored in external memory and the items that comprise these facts in external memory.
Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of "memory hops" before the answer is produced.
- Score: 16.362284088767456
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent research developing neural network architectures with external memory
have often used the benchmark bAbI question and answering dataset which
provides a challenging number of tasks requiring reasoning. Here we employed a
classic associative inference task from the memory-based reasoning neuroscience
literature in order to more carefully probe the reasoning capacity of existing
memory-augmented architectures. This task is thought to capture the essence of
reasoning -- the appreciation of distant relationships among elements
distributed across multiple facts or memories. Surprisingly, we found that
current architectures struggle to reason over long distance associations.
Similar results were obtained on a more complex task involving finding the
shortest path between nodes in a path. We therefore developed MEMO, an
architecture endowed with the capacity to reason over longer distances. This
was accomplished with the addition of two novel components. First, it
introduces a separation between memories (facts) stored in external memory and
the items that comprise these facts in external memory. Second, it makes use of
an adaptive retrieval mechanism, allowing a variable number of "memory hops"
before the answer is produced. MEMO is capable of solving our novel reasoning
tasks, as well as match state of the art results in bAbI.
Related papers
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory [68.97819665784442]
This paper introduces LongMemEval, a benchmark designed to evaluate five core long-term memory abilities of chat assistants.
LongMemEval presents a significant challenge to existing long-term memory systems.
We present a unified framework that breaks down the long-term memory design into four design choices.
arXiv Detail & Related papers (2024-10-14T17:59:44Z) - Hierarchical Working Memory and a New Magic Number [1.024113475677323]
We propose a recurrent neural network model for chunking within the framework of the synaptic theory of working memory.
Our work provides a novel conceptual and analytical framework for understanding the on-the-fly organization of information in the brain that is crucial for cognition.
arXiv Detail & Related papers (2024-08-14T16:03:47Z) - B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory [91.81390121042192]
We develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an composable module.
B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens.
arXiv Detail & Related papers (2024-07-08T18:41:01Z) - Empowering Working Memory for Large Language Model Agents [9.83467478231344]
This paper explores the potential of applying cognitive psychology's working memory frameworks to large language models (LLMs)
An innovative model is proposed incorporating a centralized Working Memory Hub and Episodic Buffer access to retain memories across episodes.
This architecture aims to provide greater continuity for nuanced contextual reasoning during intricate tasks and collaborative scenarios.
arXiv Detail & Related papers (2023-12-22T05:59:00Z) - Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term
Memory [24.464945401037056]
We propose TiM (Think-in-Memory) that enables Large Language Models to maintain an evolved memory for storing historical thoughts.
We conduct qualitative and quantitative experiments on real-world and simulated dialogues covering a wide range of topics.
arXiv Detail & Related papers (2023-11-15T06:08:35Z) - A Framework for Inference Inspired by Human Memory Mechanisms [9.408704431898279]
We propose a PMI framework that consists of perception, memory and inference components.
The memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience.
We apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets.
arXiv Detail & Related papers (2023-10-01T08:12:55Z) - LaMemo: Language Modeling with Look-Ahead Memory [50.6248714811912]
We propose Look-Ahead Memory (LaMemo) that enhances the recurrence memory by incrementally attending to the right-side tokens.
LaMemo embraces bi-directional attention and segment recurrence with an additional overhead only linearly proportional to the memory length.
Experiments on widely used language modeling benchmarks demonstrate its superiority over the baselines equipped with different types of memory.
arXiv Detail & Related papers (2022-04-15T06:11:25Z) - Memory-Guided Semantic Learning Network for Temporal Sentence Grounding [55.31041933103645]
We propose a memory-augmented network that learns and memorizes the rarely appeared content in TSG tasks.
MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module.
arXiv Detail & Related papers (2022-01-03T02:32:06Z) - Learning to Rehearse in Long Sequence Memorization [107.14601197043308]
Existing reasoning tasks often have an important assumption that the input contents can be always accessed while reasoning.
Memory augmented neural networks introduce a human-like write-read memory to compress and memorize the long input sequence in one pass.
But they have two serious drawbacks: 1) they continually update the memory from current information and inevitably forget the early contents; 2) they do not distinguish what information is important and treat all contents equally.
We propose the Rehearsal Memory to enhance long-sequence memorization by self-supervised rehearsal with a history sampler.
arXiv Detail & Related papers (2021-06-02T11:58:30Z) - Self-Attentive Associative Memory [69.40038844695917]
We propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory)
We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks.
arXiv Detail & Related papers (2020-02-10T03:27:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.