Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes

• URL: http://arxiv.org/abs/2606.20708v1
• Date: Tue, 16 Jun 2026 03:53:10 GMT
• Title: Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes
• Abstract summary: Existing frameworks measure whether simulators talk like humans against ground truth from paid participants.<n>We argue this has a structural blind spot: when goal is assigned, user's motivation is endogenous, latent, and decaying.<n>We introduce decision fidelity -- whether a simulated population reproduces the decision dynamics of real users facing real consequential choices.
• Score: 6.1260812874439345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-as-user-simulation has become core infrastructure for conversational AI: agent benchmarks (tau-bench), training pipelines, and a growing body of fidelity studies all rely on LLMs role-playing the human side of dialogue. Existing frameworks measure communicative fidelity -- whether simulators talk like humans -- against ground truth from paid participants role-playing assigned goals. We argue this has a structural blind spot: when the goal is assigned, the user's willingness is exogenous, so no framework can test whether simulators make decisions like real users whose motivation is endogenous, latent, and decaying. We introduce decision fidelity -- whether a simulated population reproduces the decision-state dynamics of real users facing real, consequential choices -- and measure it on a unique testbed: 2,790 production conversations between an LLM sales agent and real customers, including 793 with verified payment outcomes. Using a teacher-forced probe protocol that holds context and instrument fixed, we find a systematic, outcome-correlated failure we call the disengagement deficit: simulators reproduce eventual buyers almost exactly (depth bias +0.09) but inflate eventual non-buyers toward the purchase frame (depth bias +0.40; d=0.38, p<0.001), halving expressed resistance (25.1% to 13.5%) and nearly doubling deliberation (21.9% to 40.1%) while fabricating no purchases. The deficit replicates across model families (DeepSeek: d=0.41, p=0.002) and resists the obvious fix: instructing the simulator that it may disengage cuts marginal bias five-fold but barely moves the outcome-conditioned contrast (d=0.34, p=0.008). Real non-buyers say "not now" and stop; simulated non-buyers ask about price. Evaluating or training sales and persuasion agents against such simulators overstates funnel progress exactly where it matters most -- the customers who walk away.

Related papers

• CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship [7.643253506288442]
  Existing benchmarks use hand-authored scenarios and prompted simulators.<n>We introduce CompanionBench, an interactive bilingual benchmark.<n>It is the first companion benchmark to ground both its scenarios and a trained user simulator in real-world data.
  arXiv  Detail & Related papers  (2026-08-03T10:44:04Z)
• YUKTI: From Natural-Language Situations to Robust, Verifiable Decisions An Uncertainty-Typed Proposition IR, Assumption-Robust Pareto Frontiers, and a Regret Certificate [0.0]
  We show that YUKTI routes each stage to an exact, nonlinear, or evolutionary solver.<n>We prove a bound making rho an exact factor of decision regret, add auditable traceability, and synthesize a benchmark-faithful data foundation.<n>We validate three ways: under controlled misspecification the robust compromise cuts mean and tail regret by over 90% versus a naive point plan; and on a regulated commercial decision we optimize inside a lawful action space and price the downside in euros.
  arXiv  Detail & Related papers  (2026-06-22T09:29:13Z)
• OdysSim: Building Foundation Models for Human Behavior Simulation [70.35265860287608]
  Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.<n>Yet helpfulness-driven post-training pulls them toward a homogeneous, overly agreeable assistant register.<n>We present OdysSim, the largest open systematic investigation of behavioral foundation models.
  arXiv  Detail & Related papers  (2026-06-12T07:31:55Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators [63.68151307455963]
  We present SalesSim, a framework and testbed for evaluating the ability of Multimodal Large Language Models (MLLMs) to simulate realistic, persona-driven customer behavior.<n>We measure consistency between the simulator's actions and its persona specifications, as well as conversational quality.<n>Our experiments demonstrate that UserGRPO boosts decision alignment of the baseline model by 13.8% while improving conversational quality.
  arXiv  Detail & Related papers  (2026-05-08T17:59:23Z)
• Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation [15.639936579155922]
  Simulating group-level user behavior enables scalable counterfactual evaluation of merchant strategies without costly online experiments.<n>We propose Policy-Guided Hybrid Simulation (PGHS), a dual-process framework that mines transferable decision policies from behavioral trajectories.<n>PGHS achieves a group simulation error of 8.80%, improving over the best reasoning-based and fitting-based baselines by 45.8% and 40.9% respectively.
  arXiv  Detail & Related papers  (2026-04-16T16:23:13Z)
• Sell More, Play Less: Benchmarking LLM Realistic Selling Skill [1.1560747347923173]
  SalesLLM benchmark is built from realistic applications covering Financial Services and Consumer Goods.<n>SalesLLM benchmark scores correlate strongly with expert human ratings.
  arXiv  Detail & Related papers  (2026-04-08T13:06:37Z)
• RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation [43.758401022409465]
  RealUserSim is the first user simulation framework grounded in real behavioral data.<n>From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles.<n>A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3%.
  arXiv  Detail & Related papers  (2026-04-07T19:42:36Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• HumanLM: Simulating Users with State Alignment Beats Response Imitation [84.89761487596844]
  We propose a novel training framework, HumanLM, which builds user simulators that accurately reflect real users.<n>HumanLM generates natural-language latent states that align with ground-truth responses through reinforcement learning.<n>For extensive evaluation, we develop Humanual, a comprehensive benchmark for simulating real users based on public data.
  arXiv  Detail & Related papers  (2026-02-07T20:26:28Z)
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
  We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
  arXiv  Detail & Related papers  (2025-10-20T13:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.