Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents

• URL: http://arxiv.org/abs/2606.25361v1
• Date: Wed, 24 Jun 2026 03:45:45 GMT
• Title: Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents
• Abstract summary: We present a fine-grained taxonomy of conversational memory, classify retrieved memories into different role types, and design a user-centric evaluation framework.<n>Our analysis reveal many differentiated effects of memories: clarifying memory improves responses' factual accuracy and constraint awareness, making them more correct and personalized.<n>Despite the power of frontier LLMs, these findings shed light on how different memory types can be leveraged to produce more personalized responses.
• Score: 46.28019301802407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior research on memory mechanism in RAG-based conversational system has emphasized how memory is stored and retrieved. However, far less is known about how memories with different functional roles influence response quality. Specifically, how they shape an agent's responses under varying conversational contexts and whether they lead to substantively different response behaviors. Existing evaluations in conversational system are also largely reference-based, insufficiently capturing the nuances in responses that may address users' preferences differently. In this work, we probe the impact of different memory types in shaping agents' responses. We present a fine-grained taxonomy of conversational memory, classify retrieved memories into different role types, and design a user-centric evaluation framework that simulates user perspectives. Through comparative experiments on long-term datasets and frontier LLMs, our analysis reveal many differentiated effects of memories: e.g., clarifying memory improves responses' factual accuracy and constraint awareness, making them more correct and personalized; irrelevant memory reduces topic relevance and degrades constraint awareness. Despite the power of frontier LLMs, these findings shed light on how different memory types can be leveraged to produce more personalized responses and inspire further research in this direction.

Related papers

• Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers [0.42061757959666934]
  Large language model (LLM) agents increasingly operate in settings where a single context window is too small to capture what has happened.<n>Memory is what turns a stateless text generator into a genuinely adaptive agent.<n>This survey offers a structured account of how memory is designed, implemented, and evaluated in modern LLM-based agents.
  arXiv  Detail & Related papers  (2026-03-08T15:08:01Z)
• EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory [63.84216832544323]
  EvolMem is a new benchmark for assessing multi-session memory capabilities of large language models (LLMs) and agent systems.<n>To construct the benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations.<n>Extensive evaluation reveals that no LLM consistently outperforms others across all memory dimensions.
  arXiv  Detail & Related papers  (2026-01-07T03:14:42Z)
• Memory in the Age of AI Agents [217.9368190980982]
  This work aims to provide an up-to-date landscape of current agent memory research.<n>We identify three dominant realizations of agent memory, namely token-level, parametric, and latent memory.<n>To support practical development, we compile a comprehensive summary of memory benchmarks and open-source frameworks.
  arXiv  Detail & Related papers  (2025-12-15T17:22:34Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior [65.70584076918679]
  Memory is a critical component in large language model (LLM)-based agents.<n>This paper studies how memory management choices impact the LLM agents' behavior, especially their long-term performance.
  arXiv  Detail & Related papers  (2025-05-21T22:35:01Z)
• In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents [70.12342024019044]
  Large Language Models (LLMs) have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information limits their effectiveness.<n>We propose Reflective Memory Management (RMM), a novel mechanism for long-term dialogue agents, integrating forward- and backward-looking reflections.<n>RMM shows more than 10% accuracy improvement over the baseline without memory management on the LongMemEval dataset.
  arXiv  Detail & Related papers  (2025-03-11T04:15:52Z)
• Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation [39.69790911626182]
  The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL)<n>The term memory'' encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities.<n>This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types.
  arXiv  Detail & Related papers  (2024-12-09T14:34:31Z)
• MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation [15.64077949677469]
  We present a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) to evaluate the effectiveness of memory-augmented dialogue systems (MADS) The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement.
  arXiv  Detail & Related papers  (2024-09-23T17:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.