UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

• URL: http://arxiv.org/abs/2608.30508v1
• Date: Mon, 31 Aug 2026 09:41:26 GMT
• Title: UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory
• Abstract summary: We introduce UtilMem, a diagnostic benchmark to evaluate memory utilization.<n>We find that strong performance on conventional factual-memory benchmarks does not reliably translate into effective memory utilization.<n>These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational memory will require architectures that explicitly support evidence integration and robustness to retrieval interference.
• Score: 32.625416782173204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall: whether a system can recover isolated facts or event-level details from prior interactions. Real-world memory use, however, often requires a more demanding capability: integrating distributed, implicit, and noisy evidence across extended interaction histories into coherent, task-oriented outputs. We call this capability memory utilization. Here, we introduce UtilMem, a diagnostic benchmark comprising 1,717 instances across five domains, designed to evaluate four underexplored aspects of memory utilization: reasoning over dense histories, identifying implicitly relevant memories, synthesizing distributed evidence into summaries, analyses, or plans, and resisting interference from semantically similar distractors. Evaluating a diverse set of retrieval-based and memory-augmented systems, we find that strong performance on conventional factual-memory benchmarks does not reliably translate into effective memory utilization. Moreover, retrieval alone is insufficient: even when relevant evidence is successfully recovered, systems frequently fail to integrate information across sessions or to distinguish useful evidence from plausible distractors. These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational memory will require architectures that explicitly support evidence integration and robustness to retrieval interference. Code is available at https://github.com/peijunallin/UtilMem.

Related papers

• MemFuse: Multi-Source Memory Fusion from Fragmented Observations [32.62990579442196]
  We introduce **MemFuseBench**, a benchmark for *multi-source memory fusion*.<n>MemFuseBench is built with a Scene-to-Sensor pipeline that synthesizes controllable scenarios into source-tagged observations, evidence-grounded questions, and adversarial distractors.<n>We propose **MemFuse**, a structured memory system that preserves source-level evidence in event-layer atomic memory and organizes related atomic events into cluster-layer fused memory within a causal graph.
  arXiv  Detail & Related papers  (2026-08-19T09:04:07Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• RaMem: Contextual Reinstatement for Long-term Agentic Memory [21.228780482661076]
  We propose Contextual Reinstatement for Agentic Memory (RaMem), a framework that turns retrieved memory fragments into contextually evidence.<n>RaMem operates through four coordinated stages: (i) evidence anchoring grounds each memory in its original episodic conditions, especially event time, mention time, session span, and participants; (ii) recall condition induction derives the evidence conditions implied by the query; (iii) validity-aware retrieval uses these conditions to prioritize context-compatible memories while retaining content-relevant candidates as fallback evidence; and (iv) context-preserved synthesis keeps the selected memories' structured context available to the generator
  arXiv  Detail & Related papers  (2026-06-22T04:41:14Z)
• REAL: A Reasoning-Enhanced Graph Framework for Long-Term Memory Management of LLMs [17.526686616588794]
  Large Language Models (LLMs) are increasingly expected to interact with users over long time horizons.<n>LLMs cannot retain all past interactions, making long-term memory management essential for storing, updating, and retrieving historical information.<n> REAL constructs long-term conversational memory as a temporal and confidence-aware directed property graph.
  arXiv  Detail & Related papers  (2026-06-09T10:53:10Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models [56.31411457917676]
  We introduce MemGuard, a type-aware memory framework that preserves functional memory boundaries during memory construction and retrieval.<n>Across hallucination and long-horizon conversation benchmarks, MemGuard improves memory reliability by up to 28.27% while retrieving up to 5.8x fewer memory tokens than prior methods.
  arXiv  Detail & Related papers  (2026-05-27T06:04:19Z)
• MemConflict: Evaluating Long-Term Memory Systems Under Memory Conflicts [19.9199366981741]
  We propose a diagnostic framework that treats memory validity as a query-conditioned fitness-for-use problem.<n>MemConflict formalizes dynamic, static, and conditional conflicts over temporal validity, factual correctness, and contextual applicability.<n>It simulates controlled long-horizon histories from structured user profiles, introduces cross-session conflicts, and injects semantically similar distractors to create competition among memory candidates.
  arXiv  Detail & Related papers  (2026-05-20T09:11:04Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks [55.145729491377374]
  Existing evaluations of agents with memory typically assess memorization and action in isolation.<n>We introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.<n> MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning.
  arXiv  Detail & Related papers  (2026-02-18T09:49:14Z)
• MemWeaver: Weaving Hybrid Memories for Traceable Long-Horizon Agentic Reasoning [26.119505362626338]
  We propose a unified memory framework that consolidates long-term agent experiences into three interconnected components.<n>MemWeaver employs a dual-channel retrieval strategy that jointly retrieves structured knowledge and supporting evidence to construct compact yet information-dense contexts.
  arXiv  Detail & Related papers  (2026-01-26T06:39:27Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.