Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

• URL: http://arxiv.org/abs/2606.28433v1
• Date: Fri, 26 Jun 2026 03:19:27 GMT
• Title: Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy
• Abstract summary: We argue that reinforcement learning researchers need to distinguish between two use cases of simulators.<n>This work is a call to the community to begin clearly distinguishing how they are using simulators in their work.
• Score: 14.228571575002762
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings. When running experiments, however, the goal of achieving high performance in the simulator can mutate into focusing exclusively on solving the simulator. To achieve high scores, researchers may adopt solutions exclusively meant for solving simulators, rather than learning while the agent is deployed outside a simulator. Solving simulators is also worthy of investigation, but it is a fundamentally different RL research question. In this paper, we argue that RL researchers need to distinguish between two use cases of simulators: solving simulators and using simulators as a proxy for learning in deployment. We first discuss how these two use-cases are importantly different, in terms of constraints on how the agent can use the simulator, which algorithms are appropriate, and which evaluation metrics are appropriate. We then highlight several issues and misleading conclusions that can occur by not making the distinction between these two settings clear, supported with examples and simple experiments. This work is a call to the community to begin clearly distinguishing how they are using simulators in their work, hopefully sparking further discussion on which empirical practices work best in each setting.

Related papers

• Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations [16.24449973143784]
  Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs.<n>We replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs.<n>We find that simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations.
  arXiv  Detail & Related papers  (2026-09-08T11:21:55Z)
• One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL [75.34038228759182]
  Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior.<n>We show that this approach systematically fails to generalize, and trace the failure to simulator collapse.<n>We propose two complementary solutions, one at inference time and one at training time.
  arXiv  Detail & Related papers  (2026-08-12T16:55:50Z)
• Learning User Simulators with Turing Rewards [32.70383881352914]
  Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, and more.<n>We propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models.
  arXiv  Detail & Related papers  (2026-06-17T17:58:48Z)
• IR-SIM: A Lightweight Skill-Native Simulator for Navigation, Learning, and Benchmarking [59.6657632820611]
  IR-SIM is a skill-native navigation simulator designed for rapid scenario construction, benchmarking, and robot learning.<n> scenarios are entirely defined by YAML configuration files that specify mobile robot kinematics, geometric collision checking, LiDAR sensing, visualization, and behavior modules.<n>The experiments showcase the convenience and versatility of IR-SIM in multiple tasks.
  arXiv  Detail & Related papers  (2026-06-07T16:55:37Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? [61.07963107032645]
  Large language models (LLMs) are increasingly used in interactive applications.<n>Human evaluation remains the gold standard for assessing their performance in multi-turn conversations.<n>We introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks.
  arXiv  Detail & Related papers  (2025-10-06T23:17:44Z)
• Simulator Ensembles for Trustworthy Autonomous Driving Testing [2.2956072509238123]
  MultiSim is a novel approach to multi-simulation ADAS testing based on a search-based testing approach.<n>It identifies, on average, up to 3.4X more simulator-agnostic failing tests and higher failure rates.<n>Our results show that utilizing a surrogate model during the search retains the average number of valid failures and also improves efficiency.
  arXiv  Detail & Related papers  (2025-03-11T22:34:14Z)
• Learning Optimal Decision Making for an Industrial Truck Unloading Robot using Minimal Simulator Runs [15.208516853395453]
  Most high-fidelity robotic simulators like ours are time-consuming. For the truck unloading problem, our experiments show that a significant reduction in simulator runs can be achieved.
  arXiv  Detail & Related papers  (2021-03-13T06:22:23Z)
• A User's Guide to Calibrating Robotics Simulators [54.85241102329546]
  This paper proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms.
  arXiv  Detail & Related papers  (2020-11-17T22:24:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.