Accelerating Reinforcement Learning Training Using Simulation Surrogate Models

• URL: http://arxiv.org/abs/2605.27556v1
• Date: Tue, 26 May 2026 18:23:42 GMT
• Title: Accelerating Reinforcement Learning Training Using Simulation Surrogate Models
• Abstract summary: High-output simulation models are widely used to analyze complex systems, but their high computational cost motivates the development of cheaper surrogate models.<n>In parallel, learning reinforcement (RL) has emerged as a powerful framework for making online decisions in environments.<n>We investigate a class of surrogate models suitable for accelerating RL training in settings where the reward structure, model parameters, or system dynamics change over time and explore their interactions with simulation models and RL models.
• Score: 0.1234398109349733
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-fidelity simulation models are widely used to analyze complex stochastic systems, but their high computational cost motivates the development of cheaper surrogate models that approximate the simulation model's input-output relationship. In parallel, reinforcement learning (RL) has emerged as a powerful framework for making online decisions in stochastic environments, with increasing attention being given to the use of simulation models as training environments for RL models. We investigate a class of surrogate models suitable for accelerating RL training in settings where the reward structure, model parameters, or system dynamics change over time and explore their interactions with simulation models and RL models. Through numerical experiments on a stochastic service system modeled via discrete-event simulation, we demonstrate that leveraging surrogate models can substantially accelerate RL training and re-training.

Related papers

• Transfer Learning Across Fast- and Full-Simulation Domains in High-Energy Physics [10.078675569996992]
  We systematically study transfer learning between fast-simulated and fully simulated datasets in a realistic environment.<n>We consider three representative tasks, signal-background classification, quark-gluon jet tagging, and missing transverse energy reconstruction.<n>Models are pretrained on ATLAS-like fast simulation and adapted to CMS-like fast simulation and to fully simulated ATLAS Open Data.
  arXiv  Detail & Related papers  (2026-05-08T09:18:34Z)
• Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs [58.18140409409302]
  Large Language Models (LLMs) have made substantial strides in structured tasks through Reinforcement Learning (RL)<n>Applying RL in broader domains like chatbots and content generation presents unique challenges.<n>We show a case study of reproducing existing reward model ensemble research using embedding-based reward models.
  arXiv  Detail & Related papers  (2025-02-04T19:37:35Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs [5.488334211013093]
  We show that learning an actuated model in parallel to training the RL agent significantly reduces the total amount of required data sampled from the real system. We also show that iteratively updating the model is of major importance to avoid biases in the RL training.
  arXiv  Detail & Related papers  (2023-02-14T16:14:39Z)
• Continual learning autoencoder training for a particle-in-cell simulation via streaming [52.77024349608834]
  upcoming exascale era will provide a new generation of physics simulations with high resolution. These simulations will have a high resolution, which will impact the training of machine learning models since storing a high amount of simulation data on disk is nearly impossible. This work presents an approach that trains a neural network concurrently to a running simulation without data on a disk.
  arXiv  Detail & Related papers  (2022-11-09T09:55:14Z)
• Conditional Generative Models for Simulation of EMG During Naturalistic Movements [45.698312905115955]
  We present a conditional generative neural network trained adversarially to generate motor unit activation potential waveforms. We demonstrate the ability of such a model to predictively interpolate between a much smaller number of numerical model's outputs with a high accuracy.
  arXiv  Detail & Related papers  (2022-11-03T14:49:02Z)
• Semi-analytical Industrial Cooling System Model for Reinforcement Learning [4.272330410469061]
  We present a hybrid industrial cooling system model that embeds analytical solutions within a multi-physics simulation. The model's fidelity is evaluated against real world data from a large scale cooling system.
  arXiv  Detail & Related papers  (2022-07-26T18:19:17Z)
• Scientific multi-agent reinforcement learning for wall-models of turbulent flows [5.678337324555036]
  We introduce scientific multi-agent reinforcement learning (SciMARL) for the discovery of wall models for large-eddy simulations. The present simulations reduce by several orders of magnitude the computational cost over fully-resolved simulations.
  arXiv  Detail & Related papers  (2021-06-21T14:30:10Z)
• Deep Bayesian Active Learning for Accelerating Stochastic Simulation [74.58219903138301]
  Interactive Neural Process (INP) is a deep active learning framework for simulations and with active learning approaches. For active learning, we propose a novel acquisition function, Latent Information Gain (LIG), calculated in the latent space of NP based models. The results demonstrate STNP outperforms the baselines in the learning setting and LIG achieves the state-of-the-art for active learning.
  arXiv  Detail & Related papers  (2021-06-05T01:31:51Z)
• Hybrid modeling: Applications in real-time diagnosis [64.5040763067757]
  We outline a novel hybrid modeling approach that combines machine learning inspired models and physics-based models. We are using such models for real-time diagnosis applications.
  arXiv  Detail & Related papers  (2020-03-04T00:44:57Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.