Self-Attentive Associative Memory

• URL: http://arxiv.org/abs/2002.03519v3
• Date: Thu, 11 Jun 2020 04:56:52 GMT
• Title: Self-Attentive Associative Memory
• Authors: Hung Le, Truyen Tran and Svetha Venkatesh
• Abstract summary: We propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory) We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks.
• Score: 69.40038844695917
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory). The idea is implemented through a novel Self-attentive Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of associative memories that represent the hypothetical high-order relationships between arbitrary pairs of memory elements, through which a relational memory is constructed from an item memory. The two memories are wired into a single sequential model capable of both memorization and relational reasoning. We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks, from challenging synthetic problems to practical testbeds such as geometry, graph, reinforcement learning, and question answering.

Related papers

• Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory [72.36736686941671]
  We propose a novel framework, selfmem, for improving retrieval-augmented generation models. Selfmem iteratively employs a retrieval-augmented generator to create an unbounded memory pool and using a memory selector to choose one output as memory for the subsequent generation round. We evaluate the effectiveness of selfmem on three distinct text generation tasks.
  arXiv  Detail & Related papers  (2023-05-03T21:40:54Z)
• On the Relationship Between Variational Inference and Auto-Associative Memory [68.8204255655161]
  We study how different neural network approaches to variational inference can be applied in this framework. We evaluate the obtained algorithms on the CIFAR10 and CLEVR image datasets and compare them with other associative memory models.
  arXiv  Detail & Related papers  (2022-10-14T14:18:47Z)
• Classification and Generation of real-world data with an Associative Memory Model [0.0]
  We extend the capabilities of the basic Associative Memory Model by using a Multiple-Modality framework. By storing both the images and labels as modalities, a single Memory can be used to retrieve and complete patterns.
  arXiv  Detail & Related papers  (2022-07-11T12:51:27Z)
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning [56.450090618578]
  Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. We show that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work. We propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel.
  arXiv  Detail & Related papers  (2022-05-26T08:24:01Z)
• BayesPCN: A Continually Learnable Predictive Coding Associative Memory [15.090562171434815]
  BayesPCN is a hierarchical associative memory capable of performing continual one-shot memory writes without meta-learning. Experiments show that BayesPCN can recall corrupted i.i.d. high-dimensional data observed hundreds of "timesteps" ago without a significant drop in recall ability.
  arXiv  Detail & Related papers  (2022-05-20T02:28:11Z)
• LaMemo: Language Modeling with Look-Ahead Memory [50.6248714811912]
  We propose Look-Ahead Memory (LaMemo) that enhances the recurrence memory by incrementally attending to the right-side tokens. LaMemo embraces bi-directional attention and segment recurrence with an additional overhead only linearly proportional to the memory length. Experiments on widely used language modeling benchmarks demonstrate its superiority over the baselines equipped with different types of memory.
  arXiv  Detail & Related papers  (2022-04-15T06:11:25Z)
• Entropic Associative Memory for Manuscript Symbols [0.0]
  Manuscript symbols can be stored, recognized and retrieved from an entropic digital memory that is associative and distributed but yet declarative. We discuss the operational characteristics of the entropic associative memory for retrieving objects with both complete and incomplete information.
  arXiv  Detail & Related papers  (2022-02-17T02:29:33Z)
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory [75.65949969000596]
  Episodic and semantic memory are critical components of the human memory model. We develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation.
  arXiv  Detail & Related papers  (2021-02-20T18:40:40Z)
• Distributed Associative Memory Network with Memory Refreshing Loss [5.5792083698526405]
  We introduce a novel Distributed Associative Memory architecture (DAM) with Memory Refreshing Loss (MRL) Inspired by how the human brain works, our framework encodes data with distributed representation across multiple memory blocks. MRL enables MANN to reinforce an association between input data and task objective by reproducing input data from stored memory contents.
  arXiv  Detail & Related papers  (2020-07-21T07:34:33Z)
• MEMO: A Deep Network for Flexible Combination of Episodic Memories [16.362284088767456]
  MEMO is an architecture endowed with the capacity to reason over longer distances. First, it introduces a separation between memories stored in external memory and the items that comprise these facts in external memory. Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of "memory hops" before the answer is produced.
  arXiv  Detail & Related papers  (2020-01-29T15:56:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.