Bayesian Meta-reinforcement Learning for Traffic Signal Control
- URL: http://arxiv.org/abs/2010.00163v2
- Date: Fri, 22 Oct 2021 23:56:55 GMT
- Title: Bayesian Meta-reinforcement Learning for Traffic Signal Control
- Authors: Yayi Zou, Zhiwei Qin
- Abstract summary: We propose a novel value-based Bayesian meta-reinforcement learning framework BM-DQN to robustly speed up the learning process in new scenarios.
The experiments on restricted 2D navigation and traffic signal control show that our proposed framework adapts more quickly and robustly in new scenarios than previous methods.
- Score: 5.025654873456756
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In recent years, there has been increasing amount of interest around meta
reinforcement learning methods for traffic signal control, which have achieved
better performance compared with traditional control methods. However, previous
methods lack robustness in adaptation and stability in training process in
complex situations, which largely limits its application in real-world traffic
signal control. In this paper, we propose a novel value-based Bayesian
meta-reinforcement learning framework BM-DQN to robustly speed up the learning
process in new scenarios by utilizing well-trained prior knowledge learned from
existing scenarios. This framework is based on our proposed fast-adaptation
variation to Gradient-EM Bayesian Meta-learning and the fast-update advantage
of DQN, which allows for fast adaptation to new scenarios with continual
learning ability and robustness to uncertainty. The experiments on restricted
2D navigation and traffic signal control show that our proposed framework
adapts more quickly and robustly in new scenarios than previous methods, and
specifically, much better continual learning ability in heterogeneous
scenarios.
Related papers
- Multiplicative update rules for accelerating deep learning training and
increasing robustness [69.90473612073767]
We propose an optimization framework that fits to a wide range of machine learning algorithms and enables one to apply alternative update rules.
We claim that the proposed framework accelerates training, while leading to more robust models in contrast to traditionally used additive update rule.
arXiv Detail & Related papers (2023-07-14T06:44:43Z) - Guaranteed Conservation of Momentum for Learning Particle-based Fluid
Dynamics [96.9177297872723]
We present a novel method for guaranteeing linear momentum in learned physics simulations.
We enforce conservation of momentum with a hard constraint, which we realize via antisymmetrical continuous convolutional layers.
In combination, the proposed method allows us to increase the physical accuracy of the learned simulator substantially.
arXiv Detail & Related papers (2022-10-12T09:12:59Z) - Transferable Deep Reinforcement Learning Framework for Autonomous
Vehicles with Joint Radar-Data Communications [69.24726496448713]
We propose an intelligent optimization framework based on the Markov Decision Process (MDP) to help the AV make optimal decisions.
We then develop an effective learning algorithm leveraging recent advances of deep reinforcement learning techniques to find the optimal policy for the AV.
We show that the proposed transferable deep reinforcement learning framework reduces the obstacle miss detection probability by the AV up to 67% compared to other conventional deep reinforcement learning approaches.
arXiv Detail & Related papers (2021-05-28T08:45:37Z) - Learning and Fast Adaptation for Grid Emergency Control via Deep Meta
Reinforcement Learning [22.58070790887177]
Power systems are undergoing a significant transformation with more uncertainties, less inertia and closer to operation limits.
There is an imperative need to enhance grid emergency control to maintain system reliability and security.
Great progress has been made in developing deep reinforcement learning (DRL) based grid control solutions in recent years.
Existing DRL-based solutions have two main limitations: 1) they cannot handle well with a wide range of grid operation conditions, system parameters, and contingencies; 2) they generally lack the ability to fast adapt to new grid operation conditions, system parameters, and contingencies, limiting their applicability for real-world applications.
arXiv Detail & Related papers (2021-01-13T19:45:59Z) - MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control [54.162449208797334]
Traffic signal control aims to coordinate traffic signals across intersections to improve the traffic efficiency of a district or a city.
Deep reinforcement learning (RL) has been applied to traffic signal control recently and demonstrated promising performance where each traffic signal is regarded as an agent.
We propose a novel Meta Variationally Intrinsic Motivated (MetaVIM) RL method to learn the decentralized policy for each intersection that considers neighbor information in a latent way.
arXiv Detail & Related papers (2021-01-04T03:06:08Z) - Adaptive Gradient Method with Resilience and Momentum [120.83046824742455]
We propose an Adaptive Gradient Method with Resilience and Momentum (AdaRem)
AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient.
Our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error.
arXiv Detail & Related papers (2020-10-21T14:49:00Z) - Meta Reinforcement Learning-Based Lane Change Strategy for Autonomous
Vehicles [11.180588185127892]
Supervised learning algorithms can generalize to new environments by training on a large amount of labeled data.
It can be often impractical or cost-prohibitive to obtain sufficient data for each new environment.
We propose a meta reinforcement learning (MRL) method to improve the agent's generalization capabilities.
arXiv Detail & Related papers (2020-08-28T02:57:11Z) - Adaptive Traffic Control with Deep Reinforcement Learning: Towards
State-of-the-art and Beyond [1.3999481573773072]
We study adaptive data-guided traffic planning and control using Reinforcement Learning (RL)
We propose a novel DQN-based algorithm for Traffic Control (called TC-DQN+) as a tool for fast and more reliable traffic decision-making.
arXiv Detail & Related papers (2020-07-21T17:26:20Z) - Responsive Safety in Reinforcement Learning by PID Lagrangian Methods [74.49173841304474]
Lagrangian methods exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior.
We propose a novel Lagrange multiplier update method that utilizes derivatives of the constraint function.
We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark.
arXiv Detail & Related papers (2020-07-08T08:43:14Z) - Efficiency and Equity are Both Essential: A Generalized Traffic Signal
Controller with Deep Reinforcement Learning [25.21831641893209]
We present an approach to learning policies for signal controllers using deep reinforcement learning aiming for optimized traffic flow.
Our method uses a novel formulation of the reward function that simultaneously considers efficiency and equity.
The experimental evaluations on both simulated and real-world data demonstrate that our proposed algorithm achieves state-of-the-art performance.
arXiv Detail & Related papers (2020-03-09T11:34:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.