Bridging the Reality Gap of Reinforcement Learning based Traffic Signal Control using Domain Randomization and Meta Learning

• URL: http://arxiv.org/abs/2307.11357v1
• Date: Fri, 21 Jul 2023 05:17:21 GMT
• Title: Bridging the Reality Gap of Reinforcement Learning based Traffic Signal Control using Domain Randomization and Meta Learning
• Authors: Arthur M\"uller, Matthia Sabatelli
• Abstract summary: We present a comprehensive analysis of potential simulation parameters that contribute to this reality gap. We then examine two promising strategies that can bridge this gap: Domain Randomization (DR) and Model-Agnostic Meta-Learning (MAML) Our experimental results show that both DR and MAML outperform a state-of-the-art RL algorithm.
• Score: 0.7614628596146599
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) has been widely explored in Traffic Signal Control (TSC) applications, however, still no such system has been deployed in practice. A key barrier to progress in this area is the reality gap, the discrepancy that results from differences between simulation models and their real-world equivalents. In this paper, we address this challenge by first presenting a comprehensive analysis of potential simulation parameters that contribute to this reality gap. We then also examine two promising strategies that can bridge this gap: Domain Randomization (DR) and Model-Agnostic Meta-Learning (MAML). Both strategies were trained with a traffic simulation model of an intersection. In addition, the model was embedded in LemgoRL, a framework that integrates realistic, safety-critical requirements into the control system. Subsequently, we evaluated the performance of the two methods on a separate model of the same intersection that was developed with a different traffic simulator. In this way, we mimic the reality gap. Our experimental results show that both DR and MAML outperform a state-of-the-art RL algorithm, therefore highlighting their potential to mitigate the reality gap in RLbased TSC systems.

Related papers

• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Marginalized Importance Sampling for Off-Environment Policy Evaluation [13.824507564510503]
  Reinforcement Learning (RL) methods are typically sample-inefficient, making it challenging to train and deploy RL-policies in real world robots. This paper proposes a new approach to evaluate the real-world performance of agent policies prior to deploying them in the real world. Our approach incorporates a simulator along with real-world offline data to evaluate the performance of any policy.
  arXiv  Detail & Related papers  (2023-09-04T20:52:04Z)
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation [53.85002640149283]
  Key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge. This study identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models.
  arXiv  Detail & Related papers  (2023-09-01T19:29:53Z)
• Uncertainty-aware Grounded Action Transformation towards Sim-to-Real Transfer for Traffic Signal Control [3.6216188129806643]
  We propose a simulation-to-real-world (sim-to-real) transfer approach called UGAT. We evaluate our method on a simulated traffic environment and show that it significantly improves the performance of the transferred RL policy in the real world.
  arXiv  Detail & Related papers  (2023-07-23T17:35:49Z)
• A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving [0.0]
  Deep Reinforcement Learning (DRL) has shown remarkable success in solving complex tasks. transferring DRL agents to the real world is still challenging due to the significant discrepancies between simulation and reality. We propose a robust DRL framework that leverages platform-dependent perception modules to extract task-relevant information.
  arXiv  Detail & Related papers  (2023-04-14T07:55:07Z)
• NeurIPS 2022 Competition: Driving SMARTS [60.948652154552136]
  Driving SMARTS is a regular competition designed to tackle problems caused by the distribution shift in dynamic interaction contexts. The proposed competition supports methodologically diverse solutions, such as reinforcement learning (RL) and offline learning methods.
  arXiv  Detail & Related papers  (2022-11-14T17:10:53Z)
• Sim2real for Reinforcement Learning Driven Next Generation Networks [4.29590751118341]
  Reinforcement Learning (RL) models are regarded as the key to solving RAN-related multi-objective optimization problems. One of the main reasons is the modelling gap between the simulation and the real environment, which could make the RL agent trained by simulation ill-equipped for the real environment. This article brings to the fore the sim2real challenge within the context of Open RAN (O-RAN) Several use cases are presented to exemplify and demonstrate failure modes of the simulations trained RL model in real environments.
  arXiv  Detail & Related papers  (2022-06-08T12:40:24Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)
• From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning [69.23334811890919]
  Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
  arXiv  Detail & Related papers  (2020-05-13T14:22:20Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.