A Robust and Constrained Multi-Agent Reinforcement Learning Electric Vehicle Rebalancing Method in AMoD Systems

• URL: http://arxiv.org/abs/2209.08230v2
• Date: Wed, 27 Sep 2023 16:25:22 GMT
• Title: A Robust and Constrained Multi-Agent Reinforcement Learning Electric Vehicle Rebalancing Method in AMoD Systems
• Authors: Sihong He, Yue Wang, Shuo Han, Shaofeng Zou, Fei Miao
• Abstract summary: Electric vehicles (EVs) play critical roles in autonomous mobility-on-demand (AMoD) systems. Their unique charging patterns increase the model uncertainties in AMoD systems. Model uncertainties have not been considered explicitly in EV AMoD system rebalancing.
• Score: 20.75789597995344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Electric vehicles (EVs) play critical roles in autonomous mobility-on-demand (AMoD) systems, but their unique charging patterns increase the model uncertainties in AMoD systems (e.g. state transition probability). Since there usually exists a mismatch between the training and test/true environments, incorporating model uncertainty into system design is of critical importance in real-world applications. However, model uncertainties have not been considered explicitly in EV AMoD system rebalancing by existing literature yet, and the coexistence of model uncertainties and constraints that the decision should satisfy makes the problem even more challenging. In this work, we design a robust and constrained multi-agent reinforcement learning (MARL) framework with state transition kernel uncertainty for EV AMoD systems. We then propose a robust and constrained MARL algorithm (ROCOMA) with robust natural policy gradients (RNPG) that trains a robust EV rebalancing policy to balance the supply-demand ratio and the charging utilization rate across the city under model uncertainty. Experiments show that the ROCOMA can learn an effective and robust rebalancing policy. It outperforms non-robust MARL methods in the presence of model uncertainties. It increases the system fairness by 19.6% and decreases the rebalancing costs by 75.8%.

Related papers

• Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning [12.490614705930676]
  Current approaches to model-based offline Reinforcement Learning (RL) often incorporate uncertainty-based reward penalization. We argue that this penalization introduces excessive conservatism, potentially resulting in suboptimal policies through underestimation. We identify as an important cause of over-penalization the lack of a reliable uncertainty estimator capable of propagating uncertainties in the Bellman operator.
  arXiv  Detail & Related papers  (2024-06-06T13:58:41Z)
• Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning [2.9158689853305693]
  We consider a model-based reinforcement learning algorithm that infers the system dynamics from the available data and performs policy optimization on imaginary model rollouts. This approach is vulnerable to exploiting model errors which can lead to catastrophic failures on the real system. We show that better performance can be obtained with a single well-calibrated autoregressive model on the D4RL benchmark.
  arXiv  Detail & Related papers  (2024-02-05T10:18:15Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Robust Electric Vehicle Balancing of Autonomous Mobility-On-Demand System: A Multi-Agent Reinforcement Learning Approach [6.716627474314613]
  Electric autonomous vehicles (EAVs) are getting attention in future autonomous mobility-on-demand (AMoD) systems. EAVs' unique charging patterns make it challenging to accurately predict the EAVs supply in E-AMoD systems. Despite the success of reinforcement learning-based E-AMoD balancing algorithms, state uncertainties under the EV supply or mobility demand remain unexplored.
  arXiv  Detail & Related papers  (2023-07-30T13:40:42Z)
• Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [50.920911532133154]
  The intrinsic ill-posedness and ordinal-sensitive nature of monocular depth estimation (MDE) models pose major challenges to the estimation of uncertainty degree. We propose to model the uncertainty of MDE models from the perspective of the inherent probability distributions. By simply introducing additional training regularization terms, our model, with surprisingly simple formations and without requiring extra modules or multiple inferences, can provide uncertainty estimations with state-of-the-art reliability.
  arXiv  Detail & Related papers  (2023-07-19T12:11:15Z)
• Learning Over Contracting and Lipschitz Closed-Loops for Partially-Observed Nonlinear Systems (Extended Version) [1.2430809884830318]
  This paper presents a policy parameterization for learning-based control on nonlinear, partially-observed dynamical systems. We prove that the resulting Youla-REN parameterization automatically satisfies stability (contraction) and user-tunable robustness (Lipschitz) conditions. We find that the Youla-REN performs similarly to existing learning-based and optimal control methods while also ensuring stability and exhibiting improved robustness to adversarial disturbances.
  arXiv  Detail & Related papers  (2023-04-12T23:55:56Z)
• KCRL: Krasovskii-Constrained Reinforcement Learning with Guaranteed Stability in Nonlinear Dynamical Systems [66.9461097311667]
  We propose a model-based reinforcement learning framework with formal stability guarantees. The proposed method learns the system dynamics up to a confidence interval using feature representation. We show that KCRL is guaranteed to learn a stabilizing policy in a finite number of interactions with the underlying unknown system.
  arXiv  Detail & Related papers  (2022-06-03T17:27:04Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)
• Probabilistic robust linear quadratic regulators with Gaussian processes [73.0364959221845]
  Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design. We present a novel controller synthesis for linearized GP dynamics that yields robust controllers with respect to a probabilistic stability margin.
  arXiv  Detail & Related papers  (2021-05-17T08:36:18Z)
• Strictly Batch Imitation Learning by Energy-based Distribution Matching [104.33286163090179]
  Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy, implicitly learn from rollout dynamics, and operate in an entirely offline fashion.
  arXiv  Detail & Related papers  (2020-06-25T03:27:59Z)
• Model-Free Voltage Regulation of Unbalanced Distribution Network Based on Surrogate Model and Deep Reinforcement Learning [9.984416150031217]
  This paper develops a model-free approach based on the surrogate model and deep reinforcement learning (DRL) We have also extended it to deal with unbalanced three-phase scenarios.
  arXiv  Detail & Related papers  (2020-06-24T18:49:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.