Benchmarking the Personalization Capabilities of Large Language Models

• URL: http://arxiv.org/abs/2607.20471v1
• Date: Sat, 23 May 2026 18:16:54 GMT
• Title: Benchmarking the Personalization Capabilities of Large Language Models
• Abstract summary: Personalization is the act of varying a message to induce action from a specific receiver while keeping sender, channel, and time fixed.<n>We adapt the Bayesian Persuasion framework of Kamenica and Gentzkow (2011) to generative agents and instantiate the formulation in sales.<n>We observe a consistent personalization plateau and on a Fortune 100 tech cohort no model statistically separates successful from unsuccessful outreach.
• Score: 18.115699104947467
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Personalization, the act of varying a message to induce action from a specific receiver while keeping sender, channel, and time fixed, has a long tradition in psychology and marketing as a two-party problem in which sender and receiver have independent objectives. Large language models remove the bounded-inventory constraint of classical retrieval-and-ranking approaches by generating a continuum of message variants conditioned on inferred receiver state, raising the question of how well current models perform personalization in the classical sense. Existing LLM personalization benchmarks measure sender-side adaptation, in which the receiver is the same user the model is serving. The two-party question, whether a generated message induces its intended action in a third party, has been investigated only through A/B tests and small-scale human studies that cannot be re-run against a new model on demand. We adapt the Bayesian Persuasion framework of Kamenica and Gentzkow (2011) to generative agents and instantiate the formulation in sales, where receiver actions are routinely logged against the outreach that induced them. We release SDR-Bench, a public corpus of 6,279 customer success stories spanning 22 industries and approximately 200 enterprises, served through a temporally constrained simulation that prevents future-data leakage. Across frontier LLMs and deep-research agents, we observe a consistent personalization plateau and on a Fortune 100 tech cohort no model statistically separates successful from unsuccessful outreach. A field deployment with 12 professional sales representatives validates the framework, with 48 percent of model-generated content rated immediately useful and senior-expert agreement at Pearson 0.82. We release SDR-Arena and SDR-Bench publicly to support reproducible study of generative personalization at scale.

Related papers

• Delegation Asymmetry in Agentic Recommender Systems: Measuring Two-Sided Receptivity in Online Dating [0.0]
  We study the condition that users accept not only delegating conversation to an agent, but also receiving agent-mediated communication from others.<n>We develop a latent-threshold measurement model of agent receptivity based on graded response models with latent regression.<n>We discuss implications for agentic recommender design, including disclosure, opt-in mechanics, and receptivity-aware matchmaking.
  arXiv  Detail & Related papers  (2026-08-18T17:51:16Z)
• The AI Epistemic Deference Index: A Continuous Measure of Sycophancy [42.31792244964347]
  We propose the AI Epistemic Deference Index (AEDI)<n>AEDI is a continuous, unidimensional score representing how sensitive the support expressed in a model's output is to the attitude expressed in a user's prompt.<n>We deploy it on a new curated database of 500 propositions across diverse topics and 16,000 prompts varying in user attitude, testing eight prominent models.
  arXiv  Detail & Related papers  (2026-06-05T23:16:28Z)
• BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces [53.55407626944686]
  textscBehaviorBench is a benchmark for evaluating personalized decision modeling from real-world behavioral traces.<n>textscBehaviorBench reconstructs wallet-level decision histories from observed public prediction-market and on-chain records.<n>Across 2,000 evaluation wallets, the benchmark contains 141,445 Belief instances and 1,485,972 Trade instances.
  arXiv  Detail & Related papers  (2026-06-01T19:04:36Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization [12.964518425036168]
  We introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences.<n>We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual.<n>Our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines.
  arXiv  Detail & Related papers  (2026-04-08T17:55:00Z)
• Dual Optimal: Make Your LLM Peer-like with Dignity [35.58723842086038]
  Current aligned language models exhibit a dual failure mode we term the Evasive Servant.<n>We propose the Dignified Peer framework, which counters servility with anti-sycophancy and trustworthiness.
  arXiv  Detail & Related papers  (2026-04-01T14:48:44Z)
• SpecSteer: Synergizing Local Context and Global Reasoning for Efficient Personalized Generation [79.75755802397312]
  SpecSteer is an asymmetric collaborative inference framework that synergizes private on-device context with cloud-scale reasoning.<n>Experiments demonstrate that SpecSteer successfully closes the reasoning gap and achieves superior personalized generation performance.
  arXiv  Detail & Related papers  (2026-03-17T07:51:29Z)
• HumanStudy-Bench: Towards AI Agent Design for Participant Simulation [11.906370453952265]
  Large language models (LLMs) are increasingly used as simulated participants in social science experiments.<n>We introduce HUMANSTUDY-BENCH, a benchmark and execution engine that orchestrates LLM-based agents to reconstruct human-subject experiments.<n>To evaluate fidelity at the level of scientific inference, we propose new metrics to quantify how much human and agent behaviors agree.
  arXiv  Detail & Related papers  (2026-01-31T12:07:42Z)
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection [71.8243083897721]
  Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability.<n>We present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training.
  arXiv  Detail & Related papers  (2025-09-27T10:37:11Z)
• Pathways of Thoughts: Multi-Directional Thinking for Long-form Personalized Question Answering [57.12316804290369]
  Personalization is essential for adapting question answering systems to user-specific information needs.<n>We propose Pathways of Thoughts (PoT), an inference-stage method that applies to any large language model (LLM) without requiring task-specific fine-tuning.<n>PoT consistently outperforms competitive baselines, achieving up to a 13.1% relative improvement.
  arXiv  Detail & Related papers  (2025-09-23T14:44:46Z)
• Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future [38.1810626252963]
  Self-Rewarding Language Models propose an architecture in which the Large Language Models(LLMs) both generates responses and evaluates its own outputs via LLM-as-a-Judge prompting.<n>We propose textbf Self-Rewarding Language Models that strategically coordinate past, present, and future model generations to sustain learning signals.
  arXiv  Detail & Related papers  (2025-08-08T05:25:54Z)
• Lorica: A Synergistic Fine-Tuning Framework for Advancing Personalized Adversarial Robustness [14.039639177573164]
  Mobile devices remain vulnerable to adversarial attacks, threatening model robustness and security.<n>We propose textitLorica, a personalized synergistic adversarial training framework that delivers customized defense models.<n>Experiments on benchmark datasets demonstrate that textitLorica can achieve up to 68$times$ improvements in communication efficiency.
  arXiv  Detail & Related papers  (2025-06-04T03:31:32Z)
• Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling [87.17041933863041]
  Reinforcement Learning from Human Feedback (RLHF) has achieved considerable success in aligning large language models (LLMs)<n>We introduce a $textbfR$esponse-$textbfc$onditioned $textbfB$radley-$textbfT$erry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following.<n>We also propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimization
  arXiv  Detail & Related papers  (2025-02-02T14:50:25Z)
• InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling [66.3072381478251]
  Reward hacking, also termed reward overoptimization, remains a critical challenge. We propose a framework for reward modeling, namely InfoRM, by introducing a variational information bottleneck objective. We show that InfoRM's overoptimization detection mechanism is not only effective but also robust across a broad range of datasets.
  arXiv  Detail & Related papers  (2024-02-14T17:49:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.