LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

• URL: http://arxiv.org/abs/2607.22690v2
• Date: Tue, 28 Jul 2026 04:54:09 GMT
• Title: LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
• Abstract summary: We introduce LazyMem, which deferring all memory construction to query time.<n>On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline.<n>It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline.
• Score: 10.916121108457537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.

Related papers

• LeanMem: Simple and Efficient Long-Term Memory for LLM Agents [21.43772050088654]
  We propose LeanMem, a lightweight long-term memory framework.<n>During inference, LeanMem dynamically selects memory types and allocates retrieval budgets according to query-specific evidence demands.<n>On LoCoMo and LongMemEval-S with GPT-4.1-mini and Qwen3-8B, LeanMem improves accuracy over the strongest memory-based baseline in every setting.
  arXiv  Detail & Related papers  (2026-08-04T10:59:49Z)
• Zero-Mem: Zero-Token Memory Operations for LLM Agents [17.228305588778]
  Zero-Mem preserves original interaction traces as its source of record.<n>Zero-Mem achieves competitive performance across long-memory and long-context question-answering benchmarks.
  arXiv  Detail & Related papers  (2026-07-31T13:01:06Z)
• Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents [72.3752981234916]
  Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks.<n>We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process.<n>MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5-20%.
  arXiv  Detail & Related papers  (2026-07-15T08:32:40Z)
• ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning [78.94047086588078]
  We propose Recursive Evidence as LLM Harness for Long-Context Reasoning.<n>It is a training-free inference method for improving long-context reasoning.<n> Experiments on eight long-context datasets with 128K context length show it consistently improves evidence utilization.
  arXiv  Detail & Related papers  (2026-07-02T17:59:26Z)
• PACMS: Submodular Context Selection as a Pluggable Engine for LLM Agents [0.9134244356393664]
  LLM agents operate over a context window that fills from several directions simultaneously.<n>The prevailing mechanism is recency truncation, sometimes paired with periodic summarization.<n>Retrieval augmented generation fetches external documents into the prompt but does not arbitrate the agent's emphalready-present pooled context.
  arXiv  Detail & Related papers  (2026-06-18T10:22:57Z)
• DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA [3.56156695760535]
  Large language model (LLM) agents still struggle with long-term memory question answering.<n>We present DeferMem, a long-term memory framework that decouples this problem into high-recall candidate retrieval and query-conditioned evidence distillation.<n>On LoCoMo and LongMemEval-S, DeferMem surpasses strong baselines in QA accuracy and memory-system efficiency.
  arXiv  Detail & Related papers  (2026-05-21T12:36:46Z)
• MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search [59.98962867649009]
  We propose MemSearch-o1, an agentic search framework built on reasoning-aligned memory growth and retracing.<n>MemSearch-o1 grows fine-grained memory fragments from memory seed tokens from queries, then retraces and deeply refines the memory via a contribution function, and finally reorganizes a globally connected memory path.
  arXiv  Detail & Related papers  (2026-04-19T05:35:06Z)
• Panini: Continual Learning in Token Space via Structured Memory [4.979820180013486]
  Language models are increasingly used to reason over content they were not trained on.<n>A common approach is retrieval-augmented generation (RAG), which stores verbatim documents externally (as chunks) and retrieves only a relevant subset at inference time.<n>We propose a human-like non-parametric continual learning framework, where the base model remains fixed, and learning occurs by integrating each new experience into an external semantic memory state.
  arXiv  Detail & Related papers  (2026-02-16T19:58:03Z)
• SwiftMem: Fast Agentic Memory via Query-aware Indexing [45.27116353623848]
  We propose SwiftMem, a query-aware agentic memory system that achieves sub-linear retrieval through specialized indexing over temporal and semantic dimensions.<n>Our temporal index enables logarithmic-time range queries for time-sensitive retrieval, while the semantic DAG-Tag index maps queries to relevant topics through hierarchical tag structures.<n> Experiments on LoCoMo and LongMemEval benchmarks demonstrate that SwiftMem achieves 47$times$ faster search compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-01-13T02:51:04Z)
• MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning [73.27233666920618]
  We propose MemSearcher, an agent workflow that iteratively maintains a compact memory and combines the current turn with it.<n>At each turn, MemSearcher fuses the user's question with the memory to generate reasoning traces, perform search actions, and update memory to retain only information essential for solving the task.<n>We introduce multi-context GRPO, an end-to-end RL framework that jointly optimize reasoning, search strategies, and memory management of MemSearcher Agents.
  arXiv  Detail & Related papers  (2025-11-04T18:27:39Z)
• From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents [79.87304940020256]
  Large Language Models (LLMs) have been widely adopted in conversational agents.<n>MemGAS is a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval.<n> Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks.
  arXiv  Detail & Related papers  (2025-05-26T06:13:07Z)
• Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading [63.93888816206071]
  We introduce MemWalker, a method that processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.
  arXiv  Detail & Related papers  (2023-10-08T06:18:14Z)
• Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models [30.48902594738911]
  Given a long conversation, large language models (LLMs) fail to recall past information and tend to generate inconsistent responses.<n>We propose to generate summaries/ memory using large language models (LLMs) to enhance long-term memory ability.
  arXiv  Detail & Related papers  (2023-08-29T04:59:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.