MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs

• URL: http://arxiv.org/abs/2605.08374v3
• Date: Thu, 14 May 2026 08:13:01 GMT
• Title: MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
• Abstract summary: We introduce MemQ, which propagates credit backward through a provenance DAG that records which memories were retrieved when each new memory was created.<n>Across six benchmarks, MemQ achieves the highest success rate on all six in generalization evaluation and runtime learning.<n>We further study how $$ and $$ interact with the EC-MDP structure, providing principled guidance for parameter selection and future research.
• Score: 47.66730296440261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval quality in isolation without accounting for the dependency chains through which memories enable the creation of future memories. We introduce MemQ, which applies TD($λ$) eligibility traces to memory Q-values, propagating credit backward through a provenance DAG that records which memories were retrieved when each new memory was created. Credit weight decays as $(γλ)^d$ with DAG depth $d$, replacing temporal distance with structural proximity. We formalize the setting as an Exogenous-Context MDP, whose factored transition decouples the exogenous task stream from the endogenous memory store. Across six benchmarks, spanning OS interaction, function calling, code generation, multimodal reasoning, embodied reasoning, and expert-level QA, MemQ achieves the highest success rate on all six in generalization evaluation and runtime learning, with gains largest on multi-step tasks that produce deep and relevant provenance chains (up to +5.7~pp) and smallest on single-step classification (+0.77~pp) where single-step updates already suffice. We further study how $γ$ and $λ$ interact with the EC-MDP structure, providing principled guidance for parameter selection and future research. Code is available at https://github.com/jwliao-ai/MemQ.

Related papers

• EdgeMem: LLM-Free Agent Memory Construction and Retrieval via Evidence-Preserving Multi-Anchor Hypergraph [37.27140929522461]
  Existing methods compress interaction histories into summaries or other LLM-generated representations.<n>We propose EdgeMem, an agent-memory method built around a simple principle: preserve original interaction turns and organize them through complementary content.
  arXiv  Detail & Related papers  (2026-09-03T11:17:03Z)
• Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems [5.321970297768025]
  Existing memory architectures treat all memories as equally persistent and systematically contaminate retrieved context with outdated facts.<n>We show that per-memory, type-conditioned temporal decay, can be operationalized as an auto-classified coefficient $_i$ in an external LLM-agent memory store.<n>We introduce the Temporal Generalization Test (TGT), a benchmark with held-out retention intervals and a Generalization Gap (GenGap) metric.
  arXiv  Detail & Related papers  (2026-08-05T12:12:44Z)
• Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents [72.3752981234916]
  Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks.<n>We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process.<n>MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5-20%.
  arXiv  Detail & Related papers  (2026-07-15T08:32:40Z)
• DMF: A Deterministic Memory Framework for Conversational AI Agents [0.0]
  We present the Deterministic Memory Framework (DMF), a CPU-first approach that replaces generative memory compression with a fully deterministic pipeline.<n>DMF assigns each conversational interaction a Survival Score $$ computed from deterministic content signals, conversational cues, and structured provenance, combined through a logistic projection.<n>Experiments are conducted on a purpose-built benchmark using the LoCoMo and LongMemEval datasets.
  arXiv  Detail & Related papers  (2026-06-02T10:41:28Z)
• Mem-$π$: Adaptive Memory through Learning When and What to Generate [65.31077328444077]
  We present a framework for adaptive memory in large language model (LLM) agents, where useful guidance is generated on demand.<n>Mem-$$ consistently outperforms retrieval-based and prior RL-optimized memory baselines.
  arXiv  Detail & Related papers  (2026-05-20T17:51:05Z)
• MemGym: a Long-Horizon Memory Environment for LLM Agents [69.79226770543049]
  We present MemGym, a benchmark for agentic memory that unifies existing agent gyms and in-house memory-grounded pipelines.<n>MemGym reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability.<n>Our synthetic pipelines for MEMGYM-CODEQA and MEMGYM-DR are length-controllable, ablation-verified at every stage, and tightly aligned with downstream scenarios.
  arXiv  Detail & Related papers  (2026-05-20T07:25:33Z)
• Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data [52.80567461420316]
  We propose a Memory-Augmented LLM-based Multi-Agent System (textbfMALMAS) for automated feature generation.<n>MALMAS decomposes the generation process into agents with distinct responsibilities, and a Router Agent activates an appropriate subset of agents per iteration.<n>We further integrate a memory module comprising procedural memory, feedback memory, and conceptual memory, enabling iterative refinement that adaptively guides subsequent feature generation.
  arXiv  Detail & Related papers  (2026-04-22T07:09:30Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• SEDM: Scalable Self-Evolving Distributed Memory for Agents [23.182291416527764]
  SEDM is a verifiable and adaptive framework that transforms memory from a passive repository into an active, self-optimizing component.<n>We show that SEDM improves reasoning accuracy while reducing token overhead compared with strong memory baselines.<n>Results highlight SEDM as a scalable and sustainable memory mechanism for open-ended multi-agent collaboration.
  arXiv  Detail & Related papers  (2025-09-11T14:37:37Z)
• Memento: Fine-tuning LLM Agents without Fine-tuning LLMs [36.3424780932712]
  We introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents.<n>Our method enables low-cost continual adaptation via memory-based online reinforcement learning.<n>We instantiate our agent model in the deep research setting, namely emphMemento, which attains top-1 on GAIA validation.
  arXiv  Detail & Related papers  (2025-08-22T07:25:30Z)
• From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents [79.87304940020256]
  Large Language Models (LLMs) have been widely adopted in conversational agents.<n>MemGAS is a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval.<n> Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks.
  arXiv  Detail & Related papers  (2025-05-26T06:13:07Z)
• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions [55.19217798774033]
  Memory is a fundamental component of AI systems, underpinning large language models (LLMs)-based agents.<n>In this survey, we first categorize memory representations into parametric and contextual forms.<n>We then introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression.
  arXiv  Detail & Related papers  (2025-05-01T17:31:33Z)
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory [0.5584627289325719]
  Large Language Models (LLMs) have demonstrated remarkable prowess in generating contextually coherent responses.<n>But their fixed context windows pose fundamental challenges for maintaining consistency over prolonged multi-session dialogues.<n>We introduce Mem0, a scalable memory-centric architecture that addresses this issue by dynamically extracting, consolidating, and retrieving salient information from ongoing conversations.
  arXiv  Detail & Related papers  (2025-04-28T01:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.