RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

• URL: http://arxiv.org/abs/2605.20204v1
• Date: Tue, 07 Apr 2026 19:42:36 GMT
• Title: RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
• Abstract summary: RealUserSim is the first user simulation framework grounded in real behavioral data.<n>From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles.<n>A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3%.
• Score: 43.758401022409465
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users), while hand-crafted behavioral directives trigger Directive Amplification, where models hyper-interpret instructions into unnatural behavioral extremes that vary dramatically across simulator models. We present RealUserSim, the first user simulation framework grounded in real behavioral data. From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles and use them to ground LLM simulators. A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3% across five behavioral dimensions. Agent evaluation on TauBench with 6 simulator models and extensive analysis shows that grounded simulation acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation), while Directive Amplification in existing benchmarks produces unrealistic behavior that compromises the validity of agent evaluation.

Related papers

• Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations [16.24449973143784]
  Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs.<n>We replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs.<n>We find that simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations.
  arXiv  Detail & Related papers  (2026-09-08T11:21:55Z)
• Will Scaling Improve Social Simulation with LLMs? [87.15677770205751]
  Large Language Model (LLM) social simulations are a promising research method, but they are not yet faithful enough to be adopted widely.<n>We use scaling laws to study the relationship between LLMs' compute scale, general capability benchmarks, and the fidelity of social simulation.
  arXiv  Detail & Related papers  (2026-07-02T17:30:38Z)
• OdysSim: Building Foundation Models for Human Behavior Simulation [70.35265860287608]
  Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.<n>Yet helpfulness-driven post-training pulls them toward a homogeneous, overly agreeable assistant register.<n>We present OdysSim, the largest open systematic investigation of behavioral foundation models.
  arXiv  Detail & Related papers  (2026-06-12T07:31:55Z)
• A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation [13.439555106885491]
  We study whether simulated evaluation preserves real-world conclusions in terms of policy ranking consistency, performance correlation, and perturbation-wise failure patterns.<n>This analysis allows us to characterize the limitations of existing simulators and identify what kinds of simulation signals are more aligned with real-world deployment.
  arXiv  Detail & Related papers  (2026-06-09T03:25:02Z)
• Task-Aware Automated User Profile Generation for Recommendation Simulation Using Large Language Models [8.706093337738869]
  This work proposes an Automated Profile Generation Framework for Recommendation Simulation, APG4RecSim.<n>APG4RecSim constructs realistic, coherent, and robust user profiles with minimal supervision.<n>Our results show that profiles generated by APG4RecSim are resilient to popularity- and position-induced biases.
  arXiv  Detail & Related papers  (2026-05-13T13:20:39Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation [62.51953630639423]
  Existing benchmarks lack visual realism, creating a large domain gap between simulation and reality.<n>We propose VISER, a visually realistic benchmark for evaluating robot manipulation in simulation.<n>VISER features a high-fidelity dataset of over 1,000 3D assets with physically-based rendering (PBR) materials, along with 3D scenes created from these assets through curated layouts or generation.
  arXiv  Detail & Related papers  (2026-05-07T14:13:05Z)
• DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation [2.7246995570820354]
  We present DexSim2Real, an integrated framework that bridges the sim-to-real gap for dexterous manipulation.<n>Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a
  arXiv  Detail & Related papers  (2026-05-03T17:29:29Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
  We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
  arXiv  Detail & Related papers  (2025-10-20T13:14:38Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.