All Emulators are Wrong, Many are Useful, and Some are More Useful Than Others: A Reproducible Comparison of Computer Model Surrogates

• URL: http://arxiv.org/abs/2512.09060v1
• Date: Tue, 09 Dec 2025 19:25:50 GMT
• Title: All Emulators are Wrong, Many are Useful, and Some are More Useful Than Others: A Reproducible Comparison of Computer Model Surrogates
• Abstract summary: We present a large-scale, fully reproducible comparison of $29$ distinct emulators across $60$ canonical test functions and $40$ real emulation datasets.<n>To facilitate rigorous, apples-to-apples comparisons, we introduce the R package textttduqling.<n>This framework allows researchers to compare emulators in a unified environment and makes it possible to replicate or extend previous studies with minimal effort.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accurate and efficient surrogate modeling is essential for modern computational science, and there are a staggering number of emulation methods to choose from. With new methods being developed all the time, comparing the relative strengths and weaknesses of different methods remains a challenge due to inconsistent benchmarking practices and (sometimes) limited reproducibility and transparency. In this work, we present a large-scale, fully reproducible comparison of $29$ distinct emulators across $60$ canonical test functions and $40$ real emulation datasets. To facilitate rigorous, apples-to-apples comparisons, we introduce the R package \texttt{duqling}, which streamlines reproducible simulation studies using a consistent, simple syntax, and automatic internal scaling of inputs. This framework allows researchers to compare emulators in a unified environment and makes it possible to replicate or extend previous studies with minimal effort, even across different publications. Our results provide detailed empirical insight into the strengths and weaknesses of state-of-the-art emulators and offer guidance for both method developers and practitioners selecting a surrogate for new data. We discuss best practices for emulator comparison and highlight how \texttt{duqling} can accelerate research in emulator design and application.

Related papers

• Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy [14.228571575002762]
  We argue that reinforcement learning researchers need to distinguish between two use cases of simulators.<n>This work is a call to the community to begin clearly distinguishing how they are using simulators in their work.
  arXiv  Detail & Related papers  (2026-06-26T03:19:27Z)
• SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? [61.07963107032645]
  Large language models (LLMs) are increasingly used in interactive applications.<n>Human evaluation remains the gold standard for assessing their performance in multi-turn conversations.<n>We introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks.
  arXiv  Detail & Related papers  (2025-10-06T23:17:44Z)
• Pseudo-Simulation for Autonomous Driving [66.1981253104508]
  Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations.<n>Real-world evaluation is often challenging due to safety concerns and a lack of realism.<n>Open-loop evaluation relies on metrics that generally overlook compounding errors.
  arXiv  Detail & Related papers  (2025-06-04T17:57:53Z)
• Simulator Ensembles for Trustworthy Autonomous Driving Testing [2.2956072509238123]
  MultiSim is a novel approach to multi-simulation ADAS testing based on a search-based testing approach.<n>It identifies, on average, up to 3.4X more simulator-agnostic failing tests and higher failure rates.<n>Our results show that utilizing a surrogate model during the search retains the average number of valid failures and also improves efficiency.
  arXiv  Detail & Related papers  (2025-03-11T22:34:14Z)
• An Efficient Learning Method to Connect Observables [6.165053219836395]
  We introduce a new model, the Multi parameter Eigenvalue Problem (MEP) emulator.<n>The new method connects emulators and can make predictions directly from observables to observables.<n>A simple simulation on a one-dimensional lattice confirms the performance of the MEP emulator.
  arXiv  Detail & Related papers  (2025-03-03T15:58:15Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [57.278726604424556]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers.<n>Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy.<n>We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation [98.51313127382937]
  We focus on the use of labels in the synthetic domain alone. Our approach introduces both a way to learn neural-invariant representations and a theoretically inspired view on how to sample the data from the simulator. We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data.
  arXiv  Detail & Related papers  (2021-11-15T18:37:43Z)
• Comparing Popular Simulation Environments in the Scope of Robotics and Reinforcement Learning [0.0]
  We show that the chosen simulation environments benefit the most from single core performance. Using a multi core system, multiple simulations could be run in parallel to increase the performance.
  arXiv  Detail & Related papers  (2021-03-08T09:08:53Z)
• A User's Guide to Calibrating Robotics Simulators [54.85241102329546]
  This paper proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms.
  arXiv  Detail & Related papers  (2020-11-17T22:24:26Z)
• AutoSimulate: (Quickly) Learning Synthetic Data Generation [70.82315853981838]
  We propose an efficient alternative for optimal synthetic data generation based on a novel differentiable approximation of the objective. We demonstrate that the proposed method finds the optimal data distribution faster (up to $50times$), with significantly reduced training data generation (up to $30times$) and better accuracy ($+8.7%$) on real-world test datasets than previous methods.
  arXiv  Detail & Related papers  (2020-08-16T11:36:11Z)
• Using Machine Learning to Emulate Agent-Based Simulations [0.0]
  We evaluate the performance of multiple machine-learning methods as statistical emulators for use in the analysis of agent-based models (ABMs) We propose that agent-based modelling would benefit from using machine-learning methods for emulation, as this can facilitate more robust sensitivity analyses for the models.
  arXiv  Detail & Related papers  (2020-05-05T11:48:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.