A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation

• URL: http://arxiv.org/abs/2609.08592v1
• Date: Tue, 08 Sep 2026 11:28:02 GMT
• Title: A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation
• Abstract summary: We propose a three-tier persona vector with 23 operationalized dimensions.<n>We evaluate the persona model inside a synthetic data generation pipeline across 64,698 conversations.
• Score: 0.9918136378822991
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating tool-augmented LLM agents requires diverse, realistic user inputs yet most evaluation frameworks use flat role descriptions ("you are an angry customer") that produce near-identical conversations regardless of the underlying scenario. In this paper, we propose a three-tier persona vector with 23 operationalized dimensions: 6 categorical demographics (jurisdiction, age, channel, device, language proficiency, time availability), 12 continuous behavioral traits (patience, assertiveness, digital literacy, etc.) sampled with Gaussian noise around curated profile base vectors, and 5 continuous emotional states (frustration, anxiety, trust, confidence, stress) that shift in response to scenario context. Orthogonal to the persona, a 4-level query-complexity overlay controls utterance phrasing from direct to deliberately vague. We evaluate the persona model inside a synthetic data generation pipeline across 64,698 multi-turn conversations spanning 8 named profiles and 3 production corpora. Key findings: (i) a 15.8 percentage-point spread in agent goal-achievement across personas confirms trait vectors produce measurably different user behavior; (ii) the same persona behaves differently across scenarios due to scenario-reactive emotional state shifts, validating the scenario-reactive design; (iii) domain-specific projects show persona sensitivity on booking-flow compliance (~15-20 percentage points gap between tier-aware and pressure-test personas), demonstrating the model faithfully reproduces real-world difficulty distributions; (iv) seven rule-described trait correlations produce auditable co-occurrence patterns without requiring learned covariance matrices. The persona model is fully specified for reproduction.

Related papers

• BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing [0.898478290216692]
  BLOOM-WILT elicits natural multi-turn instances of rare behaviours without training cost or access beyond the target's next-token distribution.<n>We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings.
  arXiv  Detail & Related papers  (2026-08-31T17:10:59Z)
• PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems [53.65117408512214]
  We introduce textbfPersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions.<n>Using PersonaForge, we construct a 6.3K-record training dataset and textbfPersonaForge-Bench, a manually annotated 138-task benchmark.<n>Experiments show that PersonaForge training improves the composite score by +4.1%, with gains across all four dimensions.
  arXiv  Detail & Related papers  (2026-08-28T14:33:19Z)
• Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events [34.88005801095875]
  Personality-conditioned LLM agents (PC-Agents) are increasingly used in emotional support, social simulation, and role-playing.<n>We study event-induced personality change after 11 major life events using the Big Five traits as a psychometric anchor.<n>Our results suggest that current PC-Agents simulate the mean of human personality dynamics, but not its shape.
  arXiv  Detail & Related papers  (2026-08-06T18:25:58Z)
• The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generation [0.0]
  We tested four prompt conditions (no persona, two engineer personas, and a research-librarian persona), 12 code-generation tasks, two frontier models, and five runs per cell.<n>Under pre-registered mixed-effects analysis, the condition-by-model interaction was significant for provider-reported output tokens.<n>On Claude Opus, the minimalist engineer persona reduced visible output by 30% without improving correctness, while the thorough engineer persona increased output without a correctness gain.
  arXiv  Detail & Related papers  (2026-07-19T21:47:58Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• PersonaDrift: A Benchmark for Temporal Anomaly Detection in Language-Based Dementia Monitoring [0.9668407688201359]
  PersonaDrift is a benchmark designed to evaluate machine learning and statistical methods for detecting progressive changes in daily communication.<n>The benchmark focuses on two forms of longitudinal change that caregivers highlighted as particularly salient: flattened sentiment and off-topic replies.<n>Preliminary results show that flattened sentiment can often be detected with simple statistical models in users with low baseline variability.
  arXiv  Detail & Related papers  (2025-11-20T15:15:00Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.