Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics

• URL: http://arxiv.org/abs/2209.05726v1
• Date: Tue, 13 Sep 2022 04:28:49 GMT
• Title: Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics
• Authors: C. Chen, Y. P. Huang, W. H. K. Lam, T. L. Pan, S. C. Hsu, A. Sumalee, R. X. Zhong
• Abstract summary: This work proposes an integral reinforcement learning (IRL) based approach to learning the macroscopic traffic dynamics for adaptive optimal perimeter control. To reduce the sampling complexity and use the available data more efficiently, the experience replay (ER) technique is introduced to the IRL algorithm. The convergence of the IRL-based algorithms and the stability of the controlled traffic dynamics are proven via the Lyapunov theory.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing data-driven and feedback traffic control strategies do not consider the heterogeneity of real-time data measurements. Besides, traditional reinforcement learning (RL) methods for traffic control usually converge slowly for lacking data efficiency. Moreover, conventional optimal perimeter control schemes require exact knowledge of the system dynamics and thus would be fragile to endogenous uncertainties. To handle these challenges, this work proposes an integral reinforcement learning (IRL) based approach to learning the macroscopic traffic dynamics for adaptive optimal perimeter control. This work makes the following primary contributions to the transportation literature: (a) A continuous-time control is developed with discrete gain updates to adapt to the discrete-time sensor data. (b) To reduce the sampling complexity and use the available data more efficiently, the experience replay (ER) technique is introduced to the IRL algorithm. (c) The proposed method relaxes the requirement on model calibration in a "model-free" manner that enables robustness against modeling uncertainty and enhances the real-time performance via a data-driven RL algorithm. (d) The convergence of the IRL-based algorithms and the stability of the controlled traffic dynamics are proven via the Lyapunov theory. The optimal control law is parameterized and then approximated by neural networks (NN), which moderates the computational complexity. Both state and input constraints are considered while no model linearization is required. Numerical examples and simulation experiments are presented to verify the effectiveness and efficiency of the proposed method.

Related papers

• Efficient and Robust Freeway Traffic Speed Estimation under Oblique Grid using Vehicle Trajectory Data [19.01488741469791]
  We propose an efficient and robust low-rank model for precise oblique traffic speed state estimation. The proposed method achieves up to 12% improvement in Root Mean Square Error (RMSE) in the TSE scenarios. It runs more than 20 times faster than the state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2024-11-06T15:13:40Z)
• Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control [1.5361702135159845]
  This paper introduces a knowledge-informed model-based residual reinforcement learning framework. It integrates traffic expert knowledge into a virtual environment model, employing the Intelligent Driver Model (IDM) for basic dynamics and neural networks for residual dynamics. We propose a novel strategy that combines traditional control methods with residual RL, facilitating efficient learning and policy optimization without the need to learn from scratch.
  arXiv  Detail & Related papers  (2024-08-30T16:16:57Z)
• Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization [0.0]
  We propose a Quasi-Newton training algorithm for policy optimization with a superlinear convergence rate. A simulation study illustrates that the proposed training algorithm outperforms other algorithms in terms of data efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-05-28T09:16:08Z)
• MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning [52.101643259906915]
  We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations. Existing model-based offline RL methods are not suitable for offline-to-online fine-tuning in high-dimensional domains. We propose an on-policy model-based method that can efficiently reuse prior data through model-based value expansion and policy regularization.
  arXiv  Detail & Related papers  (2024-01-06T21:04:31Z)
• Reinforcement Learning with Model Predictive Control for Highway Ramp Metering [14.389086937116582]
  This work explores the synergy between model-based and learning-based strategies to enhance traffic flow management. The control problem is formulated as an RL task by crafting a suitable stage cost function. An MPC-based RL approach, which leverages the MPC optimal problem as a function approximation for the RL algorithm, is proposed to learn to efficiently control an on-ramp.
  arXiv  Detail & Related papers  (2023-11-15T09:50:54Z)
• Large-Scale OD Matrix Estimation with A Deep Learning Method [70.78575952309023]
  The proposed method integrates deep learning and numerical optimization algorithms to infer matrix structure and guide numerical optimization. We conducted tests to demonstrate the good generalization performance of our method on a large-scale synthetic dataset.
  arXiv  Detail & Related papers  (2023-10-09T14:30:06Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)
• Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks [82.33619654835348]
  Intelligent surface (IRS) has been employed to reshape the wireless channels by controlling individual scattering elements' phase shifts. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity. In this article, we focus on machine learning (ML) approaches for performance in IRS-assisted wireless networks.
  arXiv  Detail & Related papers  (2020-08-29T08:39:43Z)
• Tracking Performance of Online Stochastic Learners [57.14673504239551]
  Online algorithms are popular in large-scale learning settings due to their ability to compute updates on the fly, without the need to store and process data in large batches. When a constant step-size is used, these algorithms also have the ability to adapt to drifts in problem parameters, such as data or model properties, and track the optimal solution with reasonable accuracy. We establish a link between steady-state performance derived under stationarity assumptions and the tracking performance of online learners under random walk models.
  arXiv  Detail & Related papers  (2020-04-04T14:16:27Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.