Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
- URL: http://arxiv.org/abs/2607.27056v2
- Date: Mon, 03 Aug 2026 12:10:34 GMT
- Title: Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
- Abstract summary: We propose Setoka, a benchmark for evaluating memory-augmented personalized agents.<n>Setoka defines four levels of user understanding, i.e., semantic memory, episodic memory, behavior pattern, and personality trait.<n>Our comprehensive evaluation reveals that while existing systems perform well on semantic memory retrieval, their performance declines on episodic memory.
- Score: 8.564332409420473
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Personalized agents are increasingly applied to assist users across a wide range of tasks. Effective personalized assistance requires not only retrieving explicit facts from past interactions stored in agent memory, but also inferring abstract personal characteristics. However, existing memory benchmarks primarily evaluate whether an agent can retrieve information explicitly stated in conversational histories, failing to provide an effective assessment of deeper user understanding. In this work, we propose Setoka, a benchmark for evaluating memory-augmented personalized agents with hierarchical user understanding from heterogeneous data. Grounded in theories from cognitive and personality psychology, Setoka defines four levels of user understanding, i.e., semantic memory, episodic memory, behavior pattern, and personality trait. Moreover, to enable realistic yet privacy-preserving evaluation, we design a psychometrics-based pipeline that synthesizes diverse, coherent heterogeneous user data and queries at scale. Finally, we leverage Setoka to evaluate 3 language models combined with 5 memory systems for 10 synthetic users. Our comprehensive evaluation reveals that while existing systems perform well on semantic memory retrieval, their performance declines on episodic memory. Moreover, when dealing with behavior pattern and personality trait understanding tasks that require integrating heterogeneous and fragmented information dispersed over time, performance declines even further. These findings demonstrate that user understanding cannot be handled by simple fact retrieval, motivating the design of memory mechanisms for cross-source integration and abstraction over long-term user behavior.
Related papers
- Graph-Based Personalized Memory for LLM Agents: Representation, Evolution, Retrieval, and Evaluation [10.79929452695567]
Large Language Model (LLM) agents are evolving from single-session tools toward long-term personal assistants.<n>This shift makes memory a core requirement for personalization, since user preferences, goals, constraints, relationships, and past experiences are accumulated gradually and often change over time.<n>Graph-based personalized memory provides a structured way to model such user information through explicit relations, temporal context, and evidence links.
arXiv Detail & Related papers (2026-09-08T11:34:04Z) - MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery [58.64329475683699]
We argue that long-term memory should be evaluated as an auditable post-interaction artifact.<n>We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users.<n>We find that successful assistance and recoverable memory behave as distinct capabilities.
arXiv Detail & Related papers (2026-06-23T13:52:46Z) - PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments [72.02445514666428]
We introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall.<n>PerMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time.<n> Experiments demonstrate that by linking related interactions, advanced memory systems can extract more precise preferences and reduce token consumption.
arXiv Detail & Related papers (2026-03-24T14:04:11Z) - AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment [54.72137309071243]
AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories.<n>We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management.<n>Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not
arXiv Detail & Related papers (2026-03-09T11:06:19Z) - Cognitive Prosthetic: An AI-Enabled Multimodal System for Episodic Recall in Knowledge Work [6.688814515265279]
This paper presents an AI-enabled proof-of-concept designed to support episodic recall in knowledge work through structured episodic capture and natural language retrieval.<n>We present CPMS as a functional proof-of-concept demonstrating the technical feasibility of transforming heterogeneous sensor data into queryable episodic memories.<n>This work contributes an end-to-end, privacy-aware architecture for AI-enabled memory augmentation in workplace settings.
arXiv Detail & Related papers (2026-03-02T16:58:53Z) - Learning Personalized Agents from Human Feedback [36.47803872623135]
We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization.<n>PAHF learns online from live interaction using explicit per-user memory.<n> benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts.
arXiv Detail & Related papers (2026-02-18T04:18:47Z) - OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents [55.27061195244624]
We formalize over-personalization into three types: Irrelevance, Repetition, and Sycophancy.<n>Agents tend to retrieve and over-attend to user memories even when unnecessary.<n>Our work takes an initial step toward more controllable and appropriate personalization in memory-augmented dialogue systems.
arXiv Detail & Related papers (2026-01-20T08:27:13Z) - Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey [211.01908189012184]
Memory, with hundreds of papers released this year, emerges as the critical solution to fill the utility gap.<n>We provide a unified view of foundation agent memory along three dimensions.<n>We then analyze how memory is instantiated and operated under different agent topologies.
arXiv Detail & Related papers (2026-01-14T07:38:38Z) - O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agents [60.1848551962911]
O-Mem is a novel memory framework based on active user profiling.<n>O-Mem supports hierarchical retrieval of persona attributes and topic-related context.
arXiv Detail & Related papers (2025-11-17T16:55:19Z) - Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization [26.34637576545121]
LLM-powered embodied agents have shown success on conventional object-rearrangement tasks.<n>However, providing personalized assistance that leverages user-specific knowledge from past interactions presents new challenges.<n>We investigate these challenges through the lens of agents' memory utilization.
arXiv Detail & Related papers (2025-05-22T08:00:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.