When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

• URL: http://arxiv.org/abs/2609.03467v1
• Date: Thu, 03 Sep 2026 07:24:33 GMT
• Title: When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents
• Abstract summary: We introduce LOCOMO-CONV, a memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed.<n>Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity.
• Score: 21.60115070942764
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.

Related papers

• CueMem: Cue-Guided Context Reconstruction for Long-Term Conversational Memory [38.76918926805768]
  CueMem is a cue-guided framework that treats extracted memory records as retrieval cues.<n>It reconstructs query-relevant dialogue context from their source turns.<n>Experiments on LoCoMo and LongMemEval show that CueMem consistently outperforms representative long-term memory baselines.
  arXiv  Detail & Related papers  (2026-09-11T02:27:22Z)
• Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering [42.31517185994942]
  RAG enables large language models to answer questions by accessing external knowledge.<n>Existing methods suffer from two key challenges: fragmented evidence scattered across temporally distant sessions, and noisy content within retrieved sessions that triggers the lost-in-the-middle effect.<n>We propose MemLoc, a unified Retrieve-Localize-Generate framework for long-term conversational memory QA.
  arXiv  Detail & Related papers  (2026-09-07T06:32:12Z)
• CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAG [17.889279174355227]
  We introduce MuMu-QA, a benchmark for multi-turn multi-hop RAG with explicit cross-turn sub-question dependency annotations, and CMT-RAG, a complementary memory framework for this setting.<n>MuMu-QA and corpus-level RAG benchmarks show that CMT-RAG consistently outperforms five categories of RAG baselines in answer accuracy.
  arXiv  Detail & Related papers  (2026-07-29T04:50:41Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• D-Mem: A Dual-Process Memory System for LLM Agents [3.5426740232689604]
  We introduce D-Mem, a dual-process memory system.<n>It retains lightweight vector retrieval for routine queries while establishing an exhaustive Full Deliberation module as a high-fidelity fallback.<n>Experiments on the LoCoMo and RealTalk benchmarks using GPT-4o-mini and Qwen3-235B-Instruct demonstrate the efficacy of our approach.
  arXiv  Detail & Related papers  (2026-03-19T08:55:22Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks [55.145729491377374]
  Existing evaluations of agents with memory typically assess memorization and action in isolation.<n>We introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.<n> MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning.
  arXiv  Detail & Related papers  (2026-02-18T09:49:14Z)
• Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents [19.76627324918285]
  We introduce textbfLoCoMo-Plus, a benchmark for assessing cognitive memory under cue--trigger semantic disconnect.<n>We show that conventional string-matching metrics and explicit task-type prompting are misaligned with such scenarios.<n> Experiments across diverse backbone models, retrieval-based methods, and memory systems demonstrate that cognitive memory remains challenging.
  arXiv  Detail & Related papers  (2026-02-11T10:22:35Z)
• AMA: Adaptive Memory via Multi-Agent Collaboration [54.490349689939166]
  We propose Adaptive Memory via Multi-Agent Collaboration (AMA), a novel framework that leverages coordinated agents to manage memory across multiple granularities.<n>AMA significantly outperforms state-of-the-art baselines while reducing token consumption by approximately 80% compared to full-context methods.
  arXiv  Detail & Related papers  (2026-01-28T08:09:49Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents [79.87304940020256]
  Large Language Models (LLMs) have been widely adopted in conversational agents.<n>MemGAS is a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval.<n> Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks.
  arXiv  Detail & Related papers  (2025-05-26T06:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.