Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

• URL: http://arxiv.org/abs/2508.10824v2
• Date: Sat, 16 Aug 2025 03:17:35 GMT
• Title: Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures
• Authors: Parsa Omidi, Xingshuai Huang, Axel Laborieux, Bahareh Nikpour, Tianyu Shi, Armaghan Eshaghi,
• Abstract summary: Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems.<n>Transformers excel at sequence modeling, but face limitations in long-range context retention, continual learning, and knowledge integration.<n>This review presents a unified framework bridging neuroscience principles, including dynamic multi-timescale memory, selective attention, and consolidation.
• Score: 4.942399246128045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems. While Transformer architectures excel at sequence modeling, they face critical limitations in long-range context retention, continual learning, and knowledge integration. This review presents a unified framework bridging neuroscience principles, including dynamic multi-timescale memory, selective attention, and consolidation, with engineering advances in Memory-Augmented Transformers. We organize recent progress through three taxonomic dimensions: functional objectives (context extension, reasoning, knowledge integration, adaptation), memory representations (parameter-encoded, state-based, explicit, hybrid), and integration mechanisms (attention fusion, gated control, associative retrieval). Our analysis of core memory operations (reading, writing, forgetting, and capacity management) reveals a shift from static caches toward adaptive, test-time learning systems. We identify persistent challenges in scalability and interference, alongside emerging solutions including hierarchical buffering and surprise-gated updates. This synthesis provides a roadmap toward cognitively-inspired, lifelong-learning Transformer architectures.

Related papers

• The AI Hippocampus: How Far are We From Human Memory? [77.04745635827278]
  Implicit memory refers to the knowledge embedded within the internal parameters of pre-trained transformers.<n>Explicit memory involves external storage and retrieval components designed to augment model outputs with dynamic, queryable knowledge representations.<n>Agentic memory introduces persistent, temporally extended memory structures within autonomous agents.
  arXiv  Detail & Related papers  (2026-01-14T03:24:08Z)
• Improving Multi-step RAG with Hypergraph-based Memory for Long-Context Complex Relational Modeling [83.29209853451697]
  Multi-step retrieval-augmented generation (RAG) has become a widely adopted strategy for enhancing large language models (LLMs)<n>We introduce HGMem, a hypergraph-based memory mechanism that extends the concept of memory into a dynamic, expressive structure for complex reasoning and global understanding.<n>In our approach, memory is represented as a hypergraph whose hyperedges correspond to distinct memory units, enabling the progressive formation of higher-order interactions within memory.
  arXiv  Detail & Related papers  (2025-12-30T03:13:10Z)
• AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents [69.39123054975218]
  Memory serves as the pivotal nexus bridging past and future.<n>Recent research on autonomous agents has increasingly focused on designing efficient memory by drawing on cognitive neuroscience.
  arXiv  Detail & Related papers  (2025-12-29T10:01:32Z)
• MemEvolve: Meta-Evolution of Agent Memory Systems [66.09735157017558]
  Self-evolving memory systems are unprecedentedly reshaping the evolutionary paradigm of large language model (LLM)-based agents.<n>MemeEvolve is a meta-evolutionary framework that jointly evolves agents' experiential knowledge and their memory architecture.<n> EvolveLab is a unified self-evolving memory that distills twelve representative memory systems into a modular design space.
  arXiv  Detail & Related papers  (2025-12-21T14:26:14Z)
• Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Tasks [29.90641701972515]
  Organisms constantly pivot between tasks such as evading predators, foraging, traversing rugged terrain, and socializing, often within milliseconds.<n>We propose a unified framework that integrates Hopfield-style associative memory modules atop a shared backbone.<n> Empirical evaluations on standard benchmarks confirm that our AM-augmented architecture significantly enhances adaptability and retention.
  arXiv  Detail & Related papers  (2025-11-30T15:45:55Z)
• PISA: A Pragmatic Psych-Inspired Unified Memory System for Enhanced AI Agency [50.712873697511206]
  Existing work often lacks adaptability to diverse tasks and overlooks the constructive and task-oriented role of AI agent memory.<n>We propose PISA, a pragmatic, psych-inspired unified memory system that treats memory as a constructive and adaptive process.<n>Our empirical evaluation, conducted on the existing LOCOMO benchmark and our newly proposed AggQA benchmark for data analysis tasks, confirms that PISA sets a new state-of-the-art by significantly enhancing adaptability and long-term knowledge retention.
  arXiv  Detail & Related papers  (2025-10-12T10:34:35Z)
• MemOS: A Memory OS for AI System [116.87568350346537]
  Large Language Models (LLMs) have become an essential infrastructure for Artificial General Intelligence (AGI)<n>Existing models mainly rely on static parameters and short-lived contextual states, limiting their ability to track user preferences or update knowledge over extended periods.<n>MemOS is a memory operating system that treats memory as a manageable system resource.
  arXiv  Detail & Related papers  (2025-07-04T17:21:46Z)
• MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models [31.944531660401722]
  We introduce MemOS, a memory operating system designed for Large Language Models (LLMs)<n>At its core is the MemCube, a standardized memory abstraction that enables tracking, fusion, and migration of heterogeneous memory.<n>MemOS establishes a memory-centric execution framework with strong controllability, adaptability, and evolvability.
  arXiv  Detail & Related papers  (2025-05-28T08:27:12Z)
• CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents [1.6082737760346446]
  Large language models (LLMs) have advanced the field of artificial intelligence (AI) and are a powerful enabler for interactive systems.<n>They still face challenges in long-term interactions that require adaptation towards the user as well as contextual knowledge and understanding of the ever-changing environment.<n>To overcome these challenges, holistic memory modeling is required to efficiently retrieve and store relevant information across interaction sessions.<n> Cognitive AI, which aims to simulate the human thought process in a computerized model, highlights interesting aspects, such as thoughts, memory mechanisms, and decision-making.
  arXiv  Detail & Related papers  (2025-05-19T12:33:52Z)
• Attention is All You Need Until You Need Retention [0.0]
  This work introduces a novel Retention Layer mechanism for Transformer based architectures, addressing their inherent lack of intrinsic retention capabilities.<n>The Retention Layer incorporates a persistent memory module capable of real time data population, dynamic recall, and guided output generation.<n>In each domain, the retention mechanism enables systems to learn incrementally, personalize outputs, and respond to evolving real world challenges effectively.
  arXiv  Detail & Related papers  (2025-01-15T21:33:53Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)
• A Framework for Inference Inspired by Human Memory Mechanisms [9.408704431898279]
  We propose a PMI framework that consists of perception, memory and inference components. The memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. We apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets.
  arXiv  Detail & Related papers  (2023-10-01T08:12:55Z)
• Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence [59.11038175596807]
  Continual learning aims to empower artificial intelligence with strong adaptability to the real world. Existing advances mainly focus on preserving memory stability to overcome catastrophic forgetting. We propose a generic approach that appropriately attenuates old memories in parameter distributions to improve learning plasticity.
  arXiv  Detail & Related papers  (2023-08-29T02:43:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.