From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

• URL: http://arxiv.org/abs/2606.08656v1
• Date: Sun, 07 Jun 2026 14:53:19 GMT
• Title: From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory
• Abstract summary: Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time is important.<n>We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions.<n>We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE)
• Score: 27.810603317768997
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.

Related papers

• AutoMem: Automated Learning of Memory as a Cognitive Skill [63.07128029793901]
  We show that memory expertise is a learned skill.<n>We introduce AutoMem, a framework that automates the two axes of memory management.<n>Our results show that memory management is an independently learnable skill.
  arXiv  Detail & Related papers  (2026-07-01T17:57:03Z)
• Rosetta Memory: Adaptive Memory for Cross-LLM Agents [16.39618390394249]
  Memory is the key component for transforming a stateless LLM into a persistent, evolving agent through experience accumulation, long-horizon planning, and continual self-improvement.<n>In practice, users frequently switch between LLMs, for example using Claude for coding and GPT for writing across tasks, or routing different steps to different backbones within a single task for cost-effective trade-offs.<n>Making upstream memory effectively adapt to and activate downstream LLMs remains a critical yet underexplored problem.<n>Specifically, we approach the above upstream-downstream memory adaptation problem from both the write and read sides, and design two profile-conditioned
  arXiv  Detail & Related papers  (2026-06-05T13:50:59Z)
• Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents [27.2861945963127]
  Memory-augmented LLM agents enable interactions that extend beyond finite context windows.<n>Training such agents with reinforcement learning in multi-session environments is challenging.<n>We introduce Memory-R2, a training framework for memory-augmented LLM agents.
  arXiv  Detail & Related papers  (2026-05-20T22:02:00Z)
• Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory [53.8372503928207]
  Large language model (LLM) agents require long-term user memory for consistent personalization.<n>Existing memory systems mainly rely on static, hand-crafted update rules.<n>We introduce MemCoE, a cognition-inspired two-stage optimization framework that learns how memory should be organized and what information to update.
  arXiv  Detail & Related papers  (2026-05-01T14:45:20Z)
• TAMTRL: Teacher-Aligned Reward Reshaping for Multi-Turn Reinforcement Learning in Long-Context Compression [24.10525956976745]
  We propose Teacher-Aligned Reward Reshaping for Multi-Turn Reinforcement Learning (TAMTRL)<n>TAMTRL leverages relevant documents as teacher signals by aligning them with each turn of model input and assigns rewards through normalized probabilities in a self-supervised manner.<n>Experiments with multiple models of varying scales across seven long-context benchmarks show that TAMTRL consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-03-23T07:42:04Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning [89.55738101744657]
  Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless.<n>We present Memory-R1, a reinforcement learning framework that equips LLMs with the ability to actively manage and utilize external memory.
  arXiv  Detail & Related papers  (2025-08-27T12:26:55Z)
• Memento: Fine-tuning LLM Agents without Fine-tuning LLMs [36.3424780932712]
  We introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents.<n>Our method enables low-cost continual adaptation via memory-based online reinforcement learning.<n>We instantiate our agent model in the deep research setting, namely emphMemento, which attains top-1 on GAIA validation.
  arXiv  Detail & Related papers  (2025-08-22T07:25:30Z)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents [84.62985963113245]
  We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with constant memory across long multi-turn tasks.<n>At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning.<n>We show that MEM1-7B improves performance by 3.5x while reducing memory usage by 3.7x compared to Qwen2.5-14B-Instruct on a 16-objective multi-hop QA task.
  arXiv  Detail & Related papers  (2025-06-18T19:44:46Z)
• RMM: Reinforced Memory Management for Class-Incremental Learning [102.20140790771265]
  Class-Incremental Learning (CIL) trains classifiers under a strict memory budget. Existing methods use a static and ad hoc strategy for memory allocation, which is often sub-optimal. We propose a dynamic memory management strategy that is optimized for the incremental phases and different object classes.
  arXiv  Detail & Related papers  (2023-01-14T00:07:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.