Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory

• URL: http://arxiv.org/abs/2605.31086v2
• Date: Mon, 01 Jun 2026 09:00:11 GMT
• Title: Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory
• Abstract summary: We introduce realistic dialogues across diverse interaction scenarios that exhibit dynamic temporal evolution and long-term coherence.<n>The resulting benchmark encompasses challenging question-answer pairs spanning seven inquiry types.<n>We identify at least one of 27 critical memory characteristics that we identify as essential yet underexplored in current research.
• Score: 54.947805187562274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In existing memory benchmarks for Large Language Models (LLMs), the evaluated dialogue sessions often lack long-term semantic consistency, and the underlying personas tend to be flat and static. Furthermore, in real-world scenarios, interactions between users and assistants involve more diverse, heterogeneous data streams, such as documents and emails. These shortcomings significantly limit the realism and effectiveness of current evaluations. To address these limitations, we introduce RHELM (Realistic, Heterogeneous, and Evolving Long-term Memory). Driven by meticulously crafted user profiles and a novel LOOP (pLan-rOllout-evOlve-Prune) module, we construct realistic dialogues across diverse interaction scenarios that exhibit dynamic temporal evolution and long-term coherence. Crucially, these dialogues are deeply integrated with heterogeneous external sources synchronized with the user's temporal event trajectory. The resulting benchmark encompasses challenging question-answer pairs spanning seven inquiry types, with each question mapping to at least one of 27 critical memory characteristics that we identify as essential yet underexplored in current research. Comprehensive experiments across full-context models, retrieval-augmented generation (RAG) methods, and representative memory frameworks reveal that contemporary approaches still expose critical weaknesses in complex, real-world settings, particularly in resolving multi-source aggregation and real-world contextual reasoning.

Related papers

• MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems [69.06764269022925]
  We study how current memory-augmented agents perform in realistic, interference-heavy, long-horizon settings.<n> MINTEval is a benchmark featuring long, highly interconnected contexts with frequently updated information.<n>Overall, MINTEval has 15.6k question-answering pairs over long-horizon contexts averaging 138.8k tokens and extending to 1.8M tokens per instance.
  arXiv  Detail & Related papers  (2026-05-18T15:43:35Z)
• PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments [72.02445514666428]
  We introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall.<n>PerMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time.<n> Experiments demonstrate that by linking related interactions, advanced memory systems can extract more precise preferences and reduce token consumption.
  arXiv  Detail & Related papers  (2026-03-24T14:04:11Z)
• MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios [33.8882826707344]
  MemGround is a rigorous long-term memory benchmark grounded in rich, gamified interactive scenarios.<n>To comprehensively quantify both memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite.
  arXiv  Detail & Related papers  (2026-03-23T02:57:39Z)
• ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support [11.480342895892404]
  Large Language Models (LLMs) have shown strong potential as conversational agents.<n>Yet, their effectiveness remains limited by deficiencies in robust long-term memory.<n> ES-MemEval is a benchmark that systematically evaluates five core memory capabilities.<n>EvoEmo is a dataset for personalized long-term emotional support.
  arXiv  Detail & Related papers  (2026-02-02T09:58:26Z)
• AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts [78.33143446024485]
  We introduce textbfAgentLongBench, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles.<n>This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios.
  arXiv  Detail & Related papers  (2026-01-28T16:05:44Z)
• RGMem: Renormalization Group-based Memory Evolution for Language Agent User Profile [8.224917568034572]
  We propose a self-evolving memory framework, inspired by the ideology of classic renormalization group (RG) in physics.<n>This framework enables to organize the dialogue history in multiple scales.<n>The core innovation of our work lies in modeling memory evolution as a multi-scale process of information compression and emergence.
  arXiv  Detail & Related papers  (2025-10-18T08:16:46Z)
• KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues [58.305425399644086]
  Multi-Turn Long-Form Question Answering (MT-LFQA) is a key application paradigm of Large Language Models (LLMs) in knowledge-intensive domains.<n>We introduce textbfKnowMT-Bench, the textitfirst-ever benchmark designed to systematically evaluate MT-LFQA for LLMs across knowledge-intensive fields.
  arXiv  Detail & Related papers  (2025-09-26T04:32:29Z)
• Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs [21.192619293355502]
  Large language models (LLMs) excel at solving problems with clear and complete statements, but often struggle with nuanced environments.<n>This benchmark provides valuable insights into the strengths and weaknesses of current LLMs in handling complex, interactive scenarios.
  arXiv  Detail & Related papers  (2025-08-13T19:14:45Z)
• NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities [51.07379913779232]
  NeedleBench is a framework for assessing retrieval and reasoning performance in long-context tasks.<n>It embeds key data points at varying depths to rigorously test model capabilities.<n>Our experiments reveal that reasoning models like Deep-R1 and OpenAI's o3 struggle with continuous retrieval and reasoning in information-dense scenarios.
  arXiv  Detail & Related papers  (2024-07-16T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.