Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learning

• URL: http://arxiv.org/abs/2601.15086v1
• Date: Wed, 21 Jan 2026 15:27:23 GMT
• Title: Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learning
• Abstract summary: Decision-making in the real world depends on memory that is both stable and adaptive.<n>Existing Reinforcement Learning benchmarks and memory-augmented agents focus primarily on retention.<n>We introduce a benchmark that explicitly tests continual memory updating under partial observability.
• Score: 44.94110361062394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective decision-making in the real world depends on memory that is both stable and adaptive: environments change over time, and agents must retain relevant information over long horizons while also updating or overwriting outdated content when circumstances shift. Existing Reinforcement Learning (RL) benchmarks and memory-augmented agents focus primarily on retention, leaving the equally critical ability of memory rewriting largely unexplored. To address this gap, we introduce a benchmark that explicitly tests continual memory updating under partial observability, i.e. the natural setting where an agent must rely on memory rather than current observations, and use it to compare recurrent, transformer-based, and structured memory architectures. Our experiments reveal that classic recurrent models, despite their simplicity, demonstrate greater flexibility and robustness in memory rewriting tasks than modern structured memories, which succeed only under narrow conditions, and transformer-based agents, which often fail beyond trivial retention cases. These findings expose a fundamental limitation of current approaches and emphasize the necessity of memory mechanisms that balance stable retention with adaptive updating. Our work highlights this overlooked challenge, introduces benchmarks to evaluate it, and offers insights for designing future RL agents with explicit and trainable forgetting mechanisms. Code: https://quartz-admirer.github.io/Memory-Rewriting/

Related papers

• Multi-Head Recurrent Memory Agents [6.282017902182834]
  Multi-Head Recurrent Memory partitions memory into independent heads governed by a stage-wise select-then-update strategy.<n>As a lightweight instantiation, we introduce Least-Recently-Updated MHM (MHM-LRU), which guarantees uniform head utilization with zero additional token overhead.
  arXiv  Detail & Related papers  (2026-07-01T22:38:54Z)
• AdMem: Advanced Memory for Task-solving Agents [13.611633561917513]
  Large Language Models (LLMs) show promise as tool-using agents but remain limited in long-horizon tasks that require remembering, organizing, and reusing knowledge.<n>We introduce a unified and automatic memory framework that integrates semantic, episodic, and procedural memory in a bi-level design combining short-term and long-term stores.
  arXiv  Detail & Related papers  (2026-06-05T00:11:57Z)
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory [50.857546269660276]
  We introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving large language models (LLMs) memory.<n>Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference.
  arXiv  Detail & Related papers  (2026-05-14T20:15:22Z)
• MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents [78.95081012334116]
  Large Language Models (LLMs) with persistent memory enhances interaction continuity and personalization but introduces new safety risks.<n> contaminated or biased memory accumulation can trigger abnormal agent behaviors.<n>MemeEvoBench is the first benchmark evaluating memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback.
  arXiv  Detail & Related papers  (2026-04-17T07:29:52Z)
• The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models [12.743453676247903]
  Large Language Models (LLMs) internalize vast world knowledge as parametric memory, yet inevitably inherit the staleness and errors of their source corpora.<n> Knowledge editing offers a pivotal paradigm for surgically modifying memory without retraining.<n>We introduce a diagnostic framework that subjects models to discriminative self-assessment under in-context learning settings.
  arXiv  Detail & Related papers  (2026-04-07T15:20:41Z)
• MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks [55.145729491377374]
  Existing evaluations of agents with memory typically assess memorization and action in isolation.<n>We introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.<n> MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning.
  arXiv  Detail & Related papers  (2026-02-18T09:49:14Z)
• Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs [31.958776309037546]
  We present Neuromem, a scalable testbed that benchmarks External Memory Modules under an interleaved insertion-and-retrieval protocol.<n>We observe that performance typically degrades as memory grows across rounds, and time-related queries remain the most challenging category.
  arXiv  Detail & Related papers  (2026-02-15T02:53:37Z)
• AMA: Adaptive Memory via Multi-Agent Collaboration [54.490349689939166]
  We propose Adaptive Memory via Multi-Agent Collaboration (AMA), a novel framework that leverages coordinated agents to manage memory across multiple granularities.<n>AMA significantly outperforms state-of-the-art baselines while reducing token consumption by approximately 80% compared to full-context methods.
  arXiv  Detail & Related papers  (2026-01-28T08:09:49Z)
• Memory in the Age of AI Agents [217.9368190980982]
  This work aims to provide an up-to-date landscape of current agent memory research.<n>We identify three dominant realizations of agent memory, namely token-level, parametric, and latent memory.<n>To support practical development, we compile a comprehensive summary of memory benchmarks and open-source frameworks.
  arXiv  Detail & Related papers  (2025-12-15T17:22:34Z)
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning [53.72709564555407]
  Memo is a transformer-based architecture and training recipe for reinforcement learning.<n>It incorporates the creation and retrieval of memory by interleaving periodic summarization tokens with the inputs of a model during training.<n>We demonstrate Memo's effectiveness on a gridworld meta-RL benchmark and a multi-object navigation task in photo-realistic indoor settings.
  arXiv  Detail & Related papers  (2025-10-22T16:24:47Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)
• Recurrent Action Transformer with Memory [39.58317527488534]
  This paper proposes a novel model architecture that incorporates a recurrent memory mechanism designed to regulate information retention. We conduct experiments on memory-intensive environments (ViZDoom-Two-Colors, T-Maze, Memory Maze, Minigrid-Memory), classic Atari games, and MuJoCo control environments. The results show that using memory can significantly improve performance in memory-intensive environments, while maintaining or improving results in classic environments.
  arXiv  Detail & Related papers  (2023-06-15T19:29:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.