Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents

• URL: http://arxiv.org/abs/2605.30159v1
• Date: Thu, 28 May 2026 16:17:19 GMT
• Title: Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
• Abstract summary: Memory-augmented LLM agents tackle complex long-horizon tasks by summarizing interaction trajectories into compact memory.<n>Existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades.<n>We argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries.
• Score: 13.846550145896314
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.

Related papers

• UniMem: Complementary Episodic-to-Parametric Memory for Boundary-Agnostic Task Streams [77.52140119830241]
  We propose UniMem, a self-routing framework for autonomous memory management.<n>UniMem uses learnable routing tokens as memory controllers, enabling adaptive coordination between complementary memory pathways.<n>Experiments on long-horizon streaming task sequences show that UniMem consistently outperforms baselines while maintaining execution fidelity.
  arXiv  Detail & Related papers  (2026-07-28T17:28:21Z)
• AttriMem: Attribution-Guided Process Feedback for Agent Memory Learning [13.898789808338634]
  We propose an attribution-guided process-feedback framework for learning memory-construction policies with RL.<n>AttriMem augments the global outcome reward with local rewards derived from token-level contributions to the final answer.
  arXiv  Detail & Related papers  (2026-07-23T09:35:34Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• Goal-Oriented Reasoning for RAG-based Memory in Conversational Agentic LLM Systems [20.461904943047468]
  Goal-Mem is a goal-oriented reasoning framework for RAG-based agentic memory.<n>We show that Goal-Mem consistently improves performance on tasks requiring multi-hop reasoning and implicit inference.
  arXiv  Detail & Related papers  (2026-05-12T14:51:02Z)
• Belief Memory: Agent Memory Under Partial Observability [56.41506249481312]
  We propose BeliefMem, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities.<n> BeliefMem preserves the uncertainty that the deterministic paradigm discards, enabling the agent to act with high confidence.<n> Empirical evaluations on LoCoMo and ALFWorld benchmarks show that, even with limited data, BeliefMem achieves the best average performance.
  arXiv  Detail & Related papers  (2026-05-07T02:03:13Z)
• MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents [78.95081012334116]
  Large Language Models (LLMs) with persistent memory enhances interaction continuity and personalization but introduces new safety risks.<n> contaminated or biased memory accumulation can trigger abnormal agent behaviors.<n>MemeEvoBench is the first benchmark evaluating memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback.
  arXiv  Detail & Related papers  (2026-04-17T07:29:52Z)
• MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning [78.46301394559903]
  Large Language Models (LLMs) are increasingly used for long-duration tasks.<n>Current methods face a trade-off between cost and accuracy.<n>MemSifter is a novel framework that offloads the memory retrieval process to a small-scale proxy model.
  arXiv  Detail & Related papers  (2026-03-03T02:57:38Z)
• From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents [78.30630000529133]
  We propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory.<n> MM-Mem memory structures hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic.<n>Experiments confirm the effectiveness of MM-Mem on both offline and streaming tasks.
  arXiv  Detail & Related papers  (2026-03-02T05:12:45Z)
• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents [57.38404718635204]
  Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows.<n>Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components.<n>We propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy.
  arXiv  Detail & Related papers  (2026-01-05T08:24:16Z)
• Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks [23.201035830828726]
  Large Language Models face challenges in long-horizon agentic tasks.<n>Existing working memory methods rely on external mechanisms that are decoupled from the agent's core policy.<n>We propose a novel framework, Memory-as-Action, where an agent actively manages its working memory by executing explicit editing operations as part of a unified policy.
  arXiv  Detail & Related papers  (2025-10-14T15:29:57Z)
• Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMs [1.2582867366903179]
  We introduce Mnemosyne, an unsupervised, human-inspired long-term memory architecture for edge-based large language models (LLM)<n>Our approach uses graph-structured storage, modular substance and redundancy filters, memory committing and pruning mechanisms, and probabilistic recall with temporal decay and refresh processes modeled after human memory.<n>Mnemosyne is designed for use in longitudinal healthcare assistants, where repetitive and semantically similar but temporally distinct conversations are limited by naive retrieval.
  arXiv  Detail & Related papers  (2025-10-07T03:32:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.