DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

• URL: http://arxiv.org/abs/2606.22877v1
• Date: Mon, 22 Jun 2026 05:41:20 GMT
• Title: DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
• Abstract summary: LLM agents increasingly act as personal assistants that must remember a user's profile over months.<n>Existing benchmarks evaluate this "memory" ability through short, simplified interactions.<n>We introduce DynamicMem, a benchmark that constructs 15 months of activity per user.
• Score: 27.335953561941736
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/

Related papers

• A Graph-Native Bitemporal Memory Store for Conversational AI Agents [0.0]
  We describe an agent-local Neo4j property graph augmented with HNSW vector indexes and a full bitemporal data model.<n>Each memory is stored as an immutable identity node linked to versioned content nodes carrying two closed-open time intervals.<n>We evaluate the system on LongMemEval, a 500-question benchmark spanning six question types designed to stress long-term memory.
  arXiv  Detail & Related papers  (2026-07-29T06:40:52Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• AdaMem: Learning What to Remember for Personalized Long-Horizon LLM Agents [79.03892269184145]
  We argue that what is worth remembering is role-dependent, and propose textbfAdaMem (Adaptive Memory)<n>AdaMem improves QA accuracy by up to textbf+9.0% over the uniform Mem0 baseline while shrinking memory volume by textbf9%.
  arXiv  Detail & Related papers  (2026-06-19T06:35:52Z)
• MemTrace: Probing What Final Accuracy Misses in Long-Term Memory [39.696350769003104]
  We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point.<n>MemTrace probes each fact along three controlled dimensions.<n>We find that similar pooled accuracy hides different failures.
  arXiv  Detail & Related papers  (2026-06-15T22:21:23Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues [80.29362825271768]
  LongMemEval-V2 is a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments.<n>LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents.<n>We propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox.
  arXiv  Detail & Related papers  (2026-05-12T17:59:34Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• LikeBench: Evaluating Subjective Likability in LLMs for Personalization [11.75597537798083]
  We argue that a third axis, likability, is both subjective and central to user experience, yet under-measured by current benchmarks.<n>We introduce LikeBench, a multi-session, dynamic evaluation framework that measures likability across multiple dimensions.<n>Our benchmark shows that strong memory performance does not guarantee high likability: DeepSeek R1, with lower memory accuracy (86%, 17 facts/profile), outperformed Qwen3 by 28% on likability score despite Qwen3's higher memory accuracy (93%, 43 facts/profile)<n>Even SOTA models like GPT-5 adapt well in short
  arXiv  Detail & Related papers  (2025-12-15T08:18:42Z)
• PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory [56.81126490418336]
  Personalization is one of the next milestones in advancing AI capability and alignment.<n> PersonaMem-v2 simulates 1,000 realistic user-chatbot interactions on 300+ scenarios, 20,000+ user preferences, and 128k-token context windows.<n>We train Qwen3-4B to outperforms GPT-5, reaching 53% accuracy in implicit personalization.
  arXiv  Detail & Related papers  (2025-12-07T06:48:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.