APeB: Benchmarking Personalization Ability of Large Language Model Agents

• URL: http://arxiv.org/abs/2607.03162v1
• Date: Fri, 03 Jul 2026 10:03:35 GMT
• Title: APeB: Benchmarking Personalization Ability of Large Language Model Agents
• Abstract summary: LLM-powered agents struggle with personalization when users issue raw, underspecified queries.<n>Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories.<n>We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories.
• Score: 17.77241512443464
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.

Related papers

• LLM4AIGQ: LLM-based AI Guidance Query Generation Framework for Multi Interest Mining [4.054665499206523]
  We propose an LLM-based solution for generating AI guidance queries tailored to users' multi-interests.<n>This approach segments user interests by integrating user profiles and historical interaction sequences, infers specific consumption intents for each sub-interest.<n>Our model achieves robust performance in both offline evaluations and online A/B tests.
  arXiv  Detail & Related papers  (2026-09-03T11:13:01Z)
• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails [42.825314356642956]
  PersonaTrail is a benchmark for personalized web agents operating in a managed open web environment.<n>PACMem is a framework that decomposes raw browsing histories into two types of structured memory.<n>Experiments show that PACMem consistently outperforms existing memory-based baselines on both tasks.
  arXiv  Detail & Related papers  (2026-05-30T13:27:55Z)
• Latent Preference Modeling for Cross-Session Personalized Tool Calling [9.4096596518028]
  We introduce MPT, a benchmark comprising 265 multi-session dialogues that cover three challenges: Preference Recall, Preference Induction, and Preference Transfer.<n>We also propose PRefine, a test-time memory-augmented method that represents user preferences as evolving hypotheses.<n>These results indicate that robust personalization in agentic systems depends on memory that captures the reasons behind user choices, not just the choices themselves.
  arXiv  Detail & Related papers  (2026-04-20T06:57:50Z)
• Towards Realistic Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions [50.70965714314064]
  Large Language Models (LLMs) are increasingly serving as personal assistants, where users share complex and diverse preferences over extended interactions.<n>This work proposes RealPref, a benchmark for evaluating realistic preference-following in personalized user-LLM interactions.
  arXiv  Detail & Related papers  (2026-03-04T15:42:43Z)
• ALPBench: A Benchmark for Attribution-level Long-term Personal Behavior Understanding [53.88804678012327]
  ALPBench is a Benchmark for Attribution-level Long-term Personal Behavior Understanding.<n>It predicts user-interested attribute combinations, enabling ground-truth evaluation.<n>It models preferences from long-term historical behaviors rather than users' explicitly expressed requests.
  arXiv  Detail & Related papers  (2026-02-03T03:32:16Z)
• Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization [27.490675380289318]
  We argue that relevance serves as an unreliable proxy for utility.<n>We propose PURPLE, a contextual bandit framework that oPtimizes UseR Profiles for Llm pErsonalization.<n>In contrast to a greedy selection of the most relevant records, PURPLE treats profile construction as a set generation process.
  arXiv  Detail & Related papers  (2026-01-17T15:05:36Z)
• LongDA: Benchmarking LLM Agents for Long-Document Data Analysis [55.32211515932351]
  LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck.<n>LongTA is a tool-augmented agent framework that enables document access, retrieval, and code execution.<n>Our experiments reveal substantial performance gaps even among state-of-the-art models.
  arXiv  Detail & Related papers  (2026-01-05T23:23:16Z)
• SessionIntentBench: A Multi-task Inter-session Intention-shift Modeling Benchmark for E-commerce Customer Behavior Understanding [64.45047674586671]
  We introduce the concept of an intention tree and propose a dataset curation pipeline.<n>We construct a sibling multimodal benchmark, SessionIntentBench, that evaluates L(V)LMs' capability on understanding inter-session intention shift.<n>With 1,952,177 intention entries, 1,132,145 session intention trajectories, and 13,003,664 available tasks mined using 10,905 sessions, we provide a scalable way to exploit the existing session data.
  arXiv  Detail & Related papers  (2025-07-27T09:04:17Z)
• From Prompting to Alignment: A Generative Framework for Query Recommendation [35.654879254147964]
  We propose a Generative Query Recommendation (GQR) framework that aligns query generation with user preference.<n>Specifically, we unify diverse query recommendation tasks by a universal prompt framework.<n>We also present a CTR-alignment framework, which involves training a query-wise CTR predictor as a process reward model.
  arXiv  Detail & Related papers  (2025-04-14T13:21:29Z)
• ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions [68.81939215223818]
  ProductAgent is a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval. We develop the agent with strategies for product feature summarization, query generation, and product retrieval. Experiments show that ProductAgent interacts positively with the user and enhances retrieval performance with increasing dialogue turns.
  arXiv  Detail & Related papers  (2024-07-01T03:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.