Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory

• URL: http://arxiv.org/abs/2605.19952v1
• Date: Tue, 19 May 2026 15:05:06 GMT
• Title: Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory
• Abstract summary: LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history.<n>We propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity.
• Score: 88.81430082035617
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To enable reliable long-term interaction, LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history. Most existing methods adopt an extracted fact based paradigm: handcrafted static prompts compress raw dialogues into atomic facts, which are then stored, matched, and injected into downstream reasoning. Nevertheless, such fact-centric designs inevitably discard fine-grained details in original dialogues and fail to support deep reasoning over scattered isolated facts. Moreover, static prompts cannot maintain consistent extraction granularity across diverse dialogue styles. To address these limitations, we propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity, extracted atomic facts for efficient memory retrieval, synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep reasoning. We further adopt TextGrad-based prompt optimization, which iteratively refines extraction and profiling prompts via response quality feedback, achieving lifelong evolution without any parameter updating. Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong memory baselines. The code is available at https://TMLR-TriMem.github.io .

Related papers

• DiaRelay: Relaying Dialogue Context with a Constant-Size Memory for Emotion Recognition in Conversation [27.460834617156028]
  Emotion Recognition in Conversation (ERC) requires models to identify subtle emotional cues that are often distributed across distant dialogue turns.<n>We propose a lightweight adapter, DiaRelay, to enable LLMs to explicitly maintain a dialogue-level memory for accurate ERC.<n>We show that DiaRelay can achieve SOTA weighted F1 and accuracy on MELD while obtaining competitive results on IEMOCAP with only an extra 7.1M trainable parameters.
  arXiv  Detail & Related papers  (2026-08-24T03:09:15Z)
• TrajWiki: Source-Grounded Memory Trajectories for Long-Horizon Dialogue Agents [17.027152676146592]
  We propose TrajWiki, a trajectory-based memory framework for long-horizon conversational agents.<n>Instead of treating memory as static entries, TrajWiki represents each memory as a source-grounded evolution trajectory.<n>TrajWiki further introduces Memory Wiki, a persistent intermediate layer that incrementally compiles dialogue history into structured and interlinked wiki pages.
  arXiv  Detail & Related papers  (2026-08-02T03:36:03Z)
• REAL: A Reasoning-Enhanced Graph Framework for Long-Term Memory Management of LLMs [17.526686616588794]
  Large Language Models (LLMs) are increasingly expected to interact with users over long time horizons.<n>LLMs cannot retain all past interactions, making long-term memory management essential for storing, updating, and retrieving historical information.<n> REAL constructs long-term conversational memory as a temporal and confidence-aware directed property graph.
  arXiv  Detail & Related papers  (2026-06-09T10:53:10Z)
• Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents [0.0]
  Existing approaches force vendor lock-in and rely on injecting large volumes of raw conversation into prompts.<n>We introduce Memori, an LLM-agnostic persistent memory layer that treats memory as a data structuring problem.
  arXiv  Detail & Related papers  (2026-03-20T13:26:38Z)
• EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory [63.84216832544323]
  EvolMem is a new benchmark for assessing multi-session memory capabilities of large language models (LLMs) and agent systems.<n>To construct the benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations.<n>Extensive evaluation reveals that no LLM consistently outperforms others across all memory dimensions.
  arXiv  Detail & Related papers  (2026-01-07T03:14:42Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• A Simple Yet Strong Baseline for Long-Term Conversational Memory of LLM Agents [4.706565675142042]
  LLM-based conversational agents still struggle to maintain coherent, personalized interaction over many sessions.<n>Motivated by neo-Davidsonian event semantics, we propose an event-centric alternative that represents conversational history as short, event-like propositions.<n>Our design aims to preserve information in a non-compressive form and make it more accessible, rather than more lossy.
  arXiv  Detail & Related papers  (2025-11-21T12:41:17Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models [30.48902594738911]
  Given a long conversation, large language models (LLMs) fail to recall past information and tend to generate inconsistent responses.<n>We propose to generate summaries/ memory using large language models (LLMs) to enhance long-term memory ability.
  arXiv  Detail & Related papers  (2023-08-29T04:59:53Z)
• SCM: Enhancing Large Language Model with Self-Controlled Memory Framework [54.33686574304374]
  Large Language Models (LLMs) are constrained by their inability to process lengthy inputs, resulting in the loss of critical historical information.<n>We propose the Self-Controlled Memory (SCM) framework to enhance the ability of LLMs to maintain long-term memory and recall relevant information.
  arXiv  Detail & Related papers  (2023-04-26T07:25:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.