Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

• URL: http://arxiv.org/abs/2605.12894v1
• Date: Wed, 13 May 2026 02:16:51 GMT
• Title: Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
• Abstract summary: Persona Policies (PPol) is a control layer that induces realistic behavioral variation in user simulators.<n>PPol produces a diverse population of human-like personas for any task in the domain.<n>Agents trained with PPol are more robust to challenging, out-of-distribution behaviors, improving task success by +17% relative to training only on existing simulated interactions.
• Score: 28.85410475505536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with a wide variety of people, including users who are unclear, impatient, or reluctant to share information. However, collecting real interaction data at scale remains expensive. The field has turned to LLM-based user simulators as stand-ins, but these simulators inherit the behavior of their underlying models: cooperative and homogeneous. As a result, agents that appear strong in simulation often fail under the unseen, diverse communication patterns of real users. To narrow this gap, we introduce Persona Policies (PPol), a plug-and-play control layer that induces realistic behavioral variation in user simulators while preserving the original task goals. Rather than hand-crafting personas, we cast persona generation as an LLM-driven evolutionary program search that optimizes a Python generator to discover behaviors and translate them into task-preserving roleplay policies. Candidate generators are guided by a multi-objective fitness score combining human-likeness with broad coverage of human behavioral patterns. Once optimized, the generator produces a diverse population of human-like personas for any task in the domain. Across tau^2-bench retail and airline domains, evolved PPol programs yield 33-62% absolute gains in fitness score over the baseline simulator. In a blinded evaluation, annotators rated PPol-conditioned users as human 80.4% of the time, close to real human traces and nearly twice as frequently as baseline simulators. Agents trained with PPol are more robust to challenging, out-of-distribution behaviors, improving task success by +17% relative to training only on existing simulated interactions. This offers a novel approach to strengthen simulator-based evaluation and training without changing tasks or rewards.

Related papers

• One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL [75.34038228759182]
  Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior.<n>We show that this approach systematically fails to generalize, and trace the failure to simulator collapse.<n>We propose two complementary solutions, one at inference time and one at training time.
  arXiv  Detail & Related papers  (2026-08-12T16:55:50Z)
• OdysSim: Building Foundation Models for Human Behavior Simulation [70.35265860287608]
  Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.<n>Yet helpfulness-driven post-training pulls them toward a homogeneous, overly agreeable assistant register.<n>We present OdysSim, the largest open systematic investigation of behavioral foundation models.
  arXiv  Detail & Related papers  (2026-06-12T07:31:55Z)
• Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants [7.523995265564992]
  We show how simulator quality can be quantified in terms of its downstream utility.<n>We train LLM assistants via reinforcement learning against a spectrum of simulators.<n>As evaluation, we measure pairwise win rates in a user study with 283 participants.
  arXiv  Detail & Related papers  (2026-05-10T23:06:24Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• HumanLM: Simulating Users with State Alignment Beats Response Imitation [84.89761487596844]
  We propose a novel training framework, HumanLM, which builds user simulators that accurately reflect real users.<n>HumanLM generates natural-language latent states that align with ground-truth responses through reinforcement learning.<n>For extensive evaluation, we develop Humanual, a comprehensive benchmark for simulating real users based on public data.
  arXiv  Detail & Related papers  (2026-02-07T20:26:28Z)
• SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? [61.07963107032645]
  Large language models (LLMs) are increasingly used in interactive applications.<n>Human evaluation remains the gold standard for assessing their performance in multi-turn conversations.<n>We introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks.
  arXiv  Detail & Related papers  (2025-10-06T23:17:44Z)
• Implicit Behavioral Alignment of Language Agents in High-Stakes Crowd Simulations [3.0112218223206173]
  Language-driven generative agents have enabled social simulations with transformative uses, from interpersonal training to aiding global policy-making.<n>Recent studies indicate that generative agent behaviors often deviate from expert expectations and real-world data--a phenomenon we term the Behavior-Realism Gap.<n>We introduce a theoretical framework called Persona-Environment Behavioral Alignment (PEBA), formulated as a distribution matching problem grounded in Lewin's behavior equation.<n>We propose PersonaEvolve (PEvo), an LLM-based optimization algorithm that iteratively refines agent personas, implicitly aligning their collective behaviors with realistic expert benchmarks within a specified environmental context.
  arXiv  Detail & Related papers  (2025-09-19T22:35:13Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.