PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

• URL: http://arxiv.org/abs/2609.09664v1
• Date: Wed, 09 Sep 2026 03:31:47 GMT
• Title: PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations
• Abstract summary: Large language models (LLMs) are increasingly deployed as personalized assistants that interact with users over extended periods of time.<n>As conversations grow longer, relying on full interaction histories becomes increasingly inefficient and unreliable.<n>We introduce PRAGMA, a benchmark for evaluating personalized guidance in long-term conversations.
• Score: 21.191446906833303
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed as personalized assistants that interact with users over extended periods of time. As conversations grow longer, relying on full interaction histories becomes increasingly inefficient and unreliable: long contexts introduce substantial computational overhead, making it difficult for models to consistently identify and utilize the most relevant information for the current request. These challenges have motivated memory systems that structure and retrieve user-specific information. In realistic interactions, users often seek practical guidance such as recommendations, planning, and decision support. Unlike factual recall tasks, personalized guidance requires models to integrate information across multiple past conversations and reason about changing user preferences and experiences. However, existing conversational memory evaluations mainly focus on retrieval and factual recall. To study this challenge, we introduce PRAGMA, a benchmark for evaluating personalized guidance in long-term conversations. PRGAMA contains curated longitudinal conversation histories, evidence annotations, and guidance scenarios grounded in evolving user contexts and incorrect user assumptions. Experiments across retrieval systems, memory systems, and long-context models reveal that current systems struggle both to recover the appropriate conversational evidence and to effectively use it for personalized guidance. Our results highlight the need for memory architectures that support robust conversational retrieval and memory-grounded reasoning beyond evidence recall.

Related papers

• Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory [15.297367578990903]
  Profile-guided Personalized Retrieval Optimization (PPRO) builds episodic and semantic memory banks from dialogue histories.<n>PPRO serves as an explicit personalized prior in memory ranking, allowing retrieval to account for stable user attributes, preferences, and relationships.<n>Experiments on LoCoMo and LongMemEval-S show consistent gains over training-free memory systems.
  arXiv  Detail & Related papers  (2026-05-28T06:47:48Z)
• PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments [72.02445514666428]
  We introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall.<n>PerMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time.<n> Experiments demonstrate that by linking related interactions, advanced memory systems can extract more precise preferences and reduce token consumption.
  arXiv  Detail & Related papers  (2026-03-24T14:04:11Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support [11.480342895892404]
  Large Language Models (LLMs) have shown strong potential as conversational agents.<n>Yet, their effectiveness remains limited by deficiencies in robust long-term memory.<n> ES-MemEval is a benchmark that systematically evaluates five core memory capabilities.<n>EvoEmo is a dataset for personalized long-term emotional support.
  arXiv  Detail & Related papers  (2026-02-02T09:58:26Z)
• OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents [55.27061195244624]
  We formalize over-personalization into three types: Irrelevance, Repetition, and Sycophancy.<n>Agents tend to retrieve and over-attend to user memories even when unnecessary.<n>Our work takes an initial step toward more controllable and appropriate personalization in memory-augmented dialogue systems.
  arXiv  Detail & Related papers  (2026-01-20T08:27:13Z)
• HiMeS: Hippocampus-inspired Memory System for Personalized AI Assistants [7.477189210398971]
  We propose HiMeS, an AI-assistant architecture that fuses short-term and long-term memory.<n>Inspired by the hippocampus-neocortex memory mechanism, we propose HiMeS, an AI-assistant architecture that fuses short-term and long-term memory.
  arXiv  Detail & Related papers  (2026-01-06T05:05:50Z)
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory [68.97819665784442]
  We introduce LongMemEval, a benchmark designed to evaluate five core long-term memory abilities of chat assistants.<n>LongMemEval presents a significant challenge to existing long-term memory systems.<n>We present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading.
  arXiv  Detail & Related papers  (2024-10-14T17:59:44Z)
• MemoCRS: Memory-enhanced Sequential Conversational Recommender Systems with Large Language Models [51.65439315425421]
  We propose a Memory-enhanced Conversational Recommender System Framework with Large Language Models (dubbed MemoCRS) User-specific memory is tailored to each user for their personalized interests. The general memory, encapsulating collaborative knowledge and reasoning guidelines, can provide shared knowledge for users.
  arXiv  Detail & Related papers  (2024-07-06T04:57:25Z)
• Sequential Recommender via Time-aware Attentive Memory Network [67.26862011527986]
  We propose a temporal gating methodology to improve attention mechanism and recurrent units. We also propose a Multi-hop Time-aware Attentive Memory network to integrate long-term and short-term preferences. Our approach is scalable for candidate retrieval tasks and can be viewed as a non-linear generalization of latent factorization for dot-product based Top-K recommendation.
  arXiv  Detail & Related papers  (2020-05-18T11:29:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.