Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling

• URL: http://arxiv.org/abs/2506.12735v1
• Date: Sun, 15 Jun 2025 06:02:42 GMT
• Title: Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling
• Authors: Zhilin Lin, Shiliang Sun,
• Abstract summary: Reinforcement learning (RL) is playing an increasingly important role in fields such as robotic control and autonomous driving.<n>The gap between simulation and the real environment remains a major obstacle to the practical deployment of RL.<n>We propose a latent space based approach to analyze the impact of simulation on real-world policy improvement.
• Score: 31.74241286023207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) is playing an increasingly important role in fields such as robotic control and autonomous driving. However, the gap between simulation and the real environment remains a major obstacle to the practical deployment of RL. Agents trained in simulators often struggle to maintain performance when transferred to real-world physical environments. In this paper, we propose a latent space based approach to analyze the impact of simulation on real-world policy improvement in model-based settings. As a natural extension of model-based methods, our approach enables an intuitive observation of the challenges faced by model-based methods in sim-to-real transfer. Experiments conducted in the MuJoCo environment evaluate the performance of our method in both measuring and mitigating the sim-to-real gap. The experiments also highlight the various challenges that remain in overcoming the sim-to-real gap, especially for model-based methods.

Related papers

• Safety-Critical Traffic Simulation with Guided Latent Diffusion Model [8.011306318131458]
  Safety-critical traffic simulation plays a crucial role in evaluating autonomous driving systems.<n>We propose a guided latent diffusion model (LDM) capable of generating physically realistic and adversarial scenarios.<n>Our work provides an effective tool for realistic safety-critical scenario simulation, paving the way for more robust evaluation of autonomous driving systems.
  arXiv  Detail & Related papers  (2025-05-01T13:33:34Z)
• From Abstraction to Reality: DARPA's Vision for Robust Sim-to-Real Autonomy [6.402441477393285]
  TIAMAT aims to address rapid and robust transfer of autonomy technologies across dynamic and complex environments.<n>Current methods for simulation-to-reality (sim-to-real) transfer often rely on high-fidelity simulations.<n>TIAMAT's approaches aim to achieve abstract-to-real transfer for effective and rapid real-world adaptation.
  arXiv  Detail & Related papers  (2025-03-14T02:06:10Z)
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics [50.191655141020505]
  This work advances model-based reinforcement learning by addressing the challenges of long-horizon prediction, error accumulation, and sim-to-real transfer.<n>By providing a scalable and robust framework, the introduced methods pave the way for adaptive and efficient robotic systems in real-world applications.
  arXiv  Detail & Related papers  (2025-01-17T10:39:09Z)
• A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning [3.1515473193934778]
  Off-dynamics Reinforcement Learning seeks to transfer a policy from a source environment to a target environment characterized by distinct yet similar dynamics. We propose an innovative approach inspired by recent advancements in Imitation Learning and conservative RL algorithms.
  arXiv  Detail & Related papers  (2023-12-24T13:09:08Z)
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation [53.85002640149283]
  Key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge. This study identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models.
  arXiv  Detail & Related papers  (2023-09-01T19:29:53Z)
• Uncertainty-aware Grounded Action Transformation towards Sim-to-Real Transfer for Traffic Signal Control [3.6216188129806643]
  We propose a simulation-to-real-world (sim-to-real) transfer approach called UGAT. We evaluate our method on a simulated traffic environment and show that it significantly improves the performance of the transferred RL policy in the real world.
  arXiv  Detail & Related papers  (2023-07-23T17:35:49Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving [26.397030011439163]
  We provide an overview of challenges with regard to different aspects and types of simulation. We cover aspects around perception-, behavior- and content-realism as well as general hurdles in the domain of simulation. Among others, we observe a trend of data-driven, generative approaches and high-fidelity data synthesis to increasingly replace model-based simulation.
  arXiv  Detail & Related papers  (2023-05-23T11:39:23Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Latent Variable Representation for Reinforcement Learning [131.03944557979725]
  It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning. We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
  arXiv  Detail & Related papers  (2022-12-17T00:26:31Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)
• From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning [69.23334811890919]
  Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
  arXiv  Detail & Related papers  (2020-05-13T14:22:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.