Disclosure-Gated User Simulation for Companion-Agent Evaluation

• URL: http://arxiv.org/abs/2609.00982v1
• Date: Tue, 01 Sep 2026 09:39:41 GMT
• Title: Disclosure-Gated User Simulation for Companion-Agent Evaluation
• Abstract summary: We use a large language model to play the user and train a user simulator against that specification.<n>Gating behaviour is learned from the training corpus's synthetic branch, while the real branch supplies how people speak and react.<n>We state two acceptance criteria: a ranking must be order-preserving, and absolute scores must be scale-stable.
• Score: 14.962552394114894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using a large language model to play the user is now standard in scalable evaluation. It has a repeatedly diagnosed failure: the simulated user is excessively cooperative, so a system under test can score by the sheer number of questions it asks rather than by making the user willing to speak. We answer with a disclosure gate conditioning information release on the companion agent's behaviour: its state is a ladder of five ordered gates, merged onto three observable depth layers. We specify, ablate, and audit it, and train a user simulator against that specification. Gating behaviour is learned from the training corpus's synthetic branch, while the real branch supplies how people speak and react; after training, the simulator need not be told at runtime which gate each item sits behind. The gate is a load-bearing component of the environment: on the English corpus of a published companion-agent benchmark (CompanionBench), once training no longer states per example which gate each item sits behind, the largest rank displacement across 12 systems under test exceeds the noise band set by re-running that environment under a new seed, while per-system scores show no detectable change. We state two acceptance criteria: a ranking must be order-preserving, and absolute scores must be scale-stable. Of the candidates we examine, only one passes both -- the simulator we release -- and its leaderboard correlates at 0.993 with the benchmark's original simulator. By contrast, prompting a frontier model as the simulator barely moves the ranking while shifting every score upward -- a shift invisible to anyone checking the ranking alone. The environment we specify is the one that benchmark already used. That publication describes the mechanism in about four hundred words, and we supply what it lacked: specification, ablations, human studies, negative controls, and downstream sensitivity analysis.

Related papers

• Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations [16.24449973143784]
  Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs.<n>We replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs.<n>We find that simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations.
  arXiv  Detail & Related papers  (2026-09-08T11:21:55Z)
• CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship [7.643253506288442]
  Existing benchmarks use hand-authored scenarios and prompted simulators.<n>We introduce CompanionBench, an interactive bilingual benchmark.<n>It is the first companion benchmark to ground both its scenarios and a trained user simulator in real-world data.
  arXiv  Detail & Related papers  (2026-08-03T10:44:04Z)
• EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management [78.71074071187753]
  We introduce EIBench, a simulator-based benchmark for interactive emotion management.<n> EIBench contains 2,222 scenarios, with 2,009 for training and 213 for held-out testing.<n>Current models perform well on support and rapport-building scenes, but struggle with boundary maintenance under user pressure.
  arXiv  Detail & Related papers  (2026-06-14T01:22:03Z)
• Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents [28.85410475505536]
  Persona Policies (PPol) is a control layer that induces realistic behavioral variation in user simulators.<n>PPol produces a diverse population of human-like personas for any task in the domain.<n>Agents trained with PPol are more robust to challenging, out-of-distribution behaviors, improving task success by +17% relative to training only on existing simulated interactions.
  arXiv  Detail & Related papers  (2026-05-13T02:16:51Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions [1.1105673928718571]
  We present SimEval-IR, an open-source toolkit and benchmark suite that makes the distinction measurable.<n>SimEval-IR provides: (1) a canonical session schema unifying session search and conversational interactions, with validated dataset adapters and explicit loss accounting; (2) three executable benchmarks covering behavioral realism, tester reliability with RATE-style estimation, and an analysis linking the two; and (3) baseline results across four real datasets in two languages and four simulator families.
  arXiv  Detail & Related papers  (2026-04-30T13:56:18Z)
• SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? [61.07963107032645]
  Large language models (LLMs) are increasingly used in interactive applications.<n>Human evaluation remains the gold standard for assessing their performance in multi-turn conversations.<n>We introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks.
  arXiv  Detail & Related papers  (2025-10-06T23:17:44Z)
• Pseudo-Simulation for Autonomous Driving [66.1981253104508]
  Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations.<n>Real-world evaluation is often challenging due to safety concerns and a lack of realism.<n>Open-loop evaluation relies on metrics that generally overlook compounding errors.
  arXiv  Detail & Related papers  (2025-06-04T17:57:53Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)
• Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems [80.77917437785773]
  Task-oriented dialogue systems ( TDSs) are assessed mainly in an offline setting or through human evaluation. We propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities.
  arXiv  Detail & Related papers  (2022-04-02T05:11:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.