SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions

• URL: http://arxiv.org/abs/2604.27878v1
• Date: Thu, 30 Apr 2026 13:56:18 GMT
• Title: SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions
• Abstract summary: We present SimEval-IR, an open-source toolkit and benchmark suite that makes the distinction measurable.<n>SimEval-IR provides: (1) a canonical session schema unifying session search and conversational interactions, with validated dataset adapters and explicit loss accounting; (2) three executable benchmarks covering behavioral realism, tester reliability with RATE-style estimation, and an analysis linking the two; and (3) baseline results across four real datasets in two languages and four simulator families.
• Score: 1.1105673928718571
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User simulators are increasingly central to interactive information retrieval, yet the community lacks standardized evaluation tools. Simulators serve two objectives, behavioral realism (matching real user behavior) and tester reliability (producing valid system rankings), and these are often conflated despite being distinct and sometimes conflicting. We present SimEval-IR, an open-source toolkit and benchmark suite that makes this distinction measurable. SimEval-IR provides: (1) a canonical session schema unifying session search and conversational interactions, with validated dataset adapters and explicit loss accounting; (2) three executable benchmarks covering behavioral realism, tester reliability with RATE-style estimation, and an analysis linking the two; and (3) baseline results across four real datasets in two languages and four simulator families. Our key finding: the classifier-discriminator ''human-likeness'' check, the dominant realism test in the literature, has essentially no pooled predictive power for system-ranking validity ($r{=}{+}0.09$, $n{=}48$), while marginal click-depth distance and Fréchet distance over session embeddings give a much stronger signal ($|r|{=}0.43$ and $0.40$, $p{\leq}0.005$). SimEval-IR is released with all configurations and scripts to reproduce the reported analysis.

Related papers

• Disclosure-Gated User Simulation for Companion-Agent Evaluation [14.962552394114894]
  We use a large language model to play the user and train a user simulator against that specification.<n>Gating behaviour is learned from the training corpus's synthetic branch, while the real branch supplies how people speak and react.<n>We state two acceptance criteria: a ranking must be order-preserving, and absolute scores must be scale-stable.
  arXiv  Detail & Related papers  (2026-09-01T09:39:41Z)
• DeepInsight II: One Trace from Benchmark to Robot [8.388561928330978]
  DeepInsight II keeps that substrate fixed and quantifies the embodied half.<n> MotionBench places four released whole-body controllers under one workload and metric contract.<n>A composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels.
  arXiv  Detail & Related papers  (2026-08-17T13:25:24Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders [48.83868690303791]
  We introduce ConvApparel, a new dataset of human-AI conversations designed to address this gap.<n>Its unique dual-agent data collection protocol -- using both "good" and "bad" recommenders -- enables counterfactual validation.<n>We propose a comprehensive validation framework that combines statistical alignment, a human-likeness score, and counterfactual validation.
  arXiv  Detail & Related papers  (2026-02-18T23:00:21Z)
• MirrorBench: An Extensible Framework to Evaluate User-Proxy Agents for Human-Likeness [0.4893345190925178]
  Large language models (LLMs) are increasingly used as human simulators.<n> naive "act-as-a-user" often yields verbose, unrealistic utterances.<n>We present MIRRORBENCH, a benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances.
  arXiv  Detail & Related papers  (2026-01-13T01:16:13Z)
• Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction [18.30483927706278]
  We present Sim4IA-Bench, a simulation benchmark suit for the prediction of the next queries and utterances.<n>Our dataset comprises 160 real-world search sessions from the CORE search engine.<n>Sim4IA-Bench provides a basis for evaluating and comparing user simu- lation approaches.
  arXiv  Detail & Related papers  (2025-11-12T13:44:12Z)
• SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? [61.07963107032645]
  Large language models (LLMs) are increasingly used in interactive applications.<n>Human evaluation remains the gold standard for assessing their performance in multi-turn conversations.<n>We introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks.
  arXiv  Detail & Related papers  (2025-10-06T23:17:44Z)
• Pseudo-Simulation for Autonomous Driving [66.1981253104508]
  Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations.<n>Real-world evaluation is often challenging due to safety concerns and a lack of realism.<n>Open-loop evaluation relies on metrics that generally overlook compounding errors.
  arXiv  Detail & Related papers  (2025-06-04T17:57:53Z)
• Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems [80.77917437785773]
  Task-oriented dialogue systems ( TDSs) are assessed mainly in an offline setting or through human evaluation. We propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities.
  arXiv  Detail & Related papers  (2022-04-02T05:11:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.