Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks

• URL: http://arxiv.org/abs/2408.06776v1
• Date: Tue, 13 Aug 2024 10:02:10 GMT
• Title: Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks
• Authors: Qiong Liu, Ye Guo, Tong Xu,
• Abstract summary: Key issue in DRL-based approaches is the limited measurement deployment in active distribution networks. To address those problems, this paper proposes a robust DRL approach with a conservative critic and a surrogate reward.
• Score: 11.073055284983626
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inverter-based volt-var control is studied in this paper. One key issue in DRL-based approaches is the limited measurement deployment in active distribution networks, which leads to problems of a partially observable state and unknown reward. To address those problems, this paper proposes a robust DRL approach with a conservative critic and a surrogate reward. The conservative critic utilizes the quantile regression technology to estimate conservative state-action value function based on the partially observable state, which helps to train a robust policy; the surrogate rewards of power loss and voltage violation are designed that can be calculated from the limited measurements. The proposed approach optimizes the power loss of the whole network and the voltage profile of buses with measurable voltages while indirectly improving the voltage profile of other buses. Extensive simulations verify the effectiveness of the robust DRL approach in different limited measurement conditions, even when only the active power injection of the root bus and less than 10% of bus voltages are measurable.

Related papers

• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization [55.97310586039358]
  Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality. We propose a novel model-free diffusion-based online RL algorithm, Q-weighted Variational Policy Optimization (QVPO) Specifically, we introduce the Q-weighted variational loss, which can be proved to be a tight lower bound of the policy objective in online RL under certain conditions. We also develop an efficient behavior policy to enhance sample efficiency by reducing the variance of the diffusion policy during online interactions.
  arXiv  Detail & Related papers  (2024-05-25T10:45:46Z)
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
  In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
  arXiv  Detail & Related papers  (2024-04-05T17:58:37Z)
• Blackout Mitigation via Physics-guided RL [17.807967857394406]
  This paper considers the sequential design of remedial control actions in response to system anomalies for the ultimate objective of preventing blackouts. A physics-guided reinforcement learning framework is designed to identify effective sequences of real-time remedial look-ahead decisions.
  arXiv  Detail & Related papers  (2024-01-17T23:27:36Z)
• Bayesian Reinforcement Learning for Automatic Voltage Control under Cyber-Induced Uncertainty [0.533024001730262]
  This work introduces a Bayesian Reinforcement Learning (BRL) approach for power system control problems. It focuses on sustained voltage control under uncertainty in a cyber-adversarial environment. BRL techniques assist in automatically finding a threshold for exploration and exploitation in various RL techniques.
  arXiv  Detail & Related papers  (2023-05-25T20:58:08Z)
• Stabilizing Voltage in Power Distribution Networks via Multi-Agent Reinforcement Learning with Transformer [128.19212716007794]
  We propose a Transformer-based Multi-Agent Actor-Critic framework (T-MAAC) to stabilize voltage in power distribution networks. In addition, we adopt a novel auxiliary-task training process tailored to the voltage control task, which improves the sample efficiency.
  arXiv  Detail & Related papers  (2022-06-08T07:48:42Z)
• RORL: Robust Offline Reinforcement Learning via Conservative Smoothing [72.8062448549897]
  offline reinforcement learning can exploit the massive amount of offline data for complex decision-making tasks. Current offline RL algorithms are generally designed to be conservative for value estimation and action selection. We propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique.
  arXiv  Detail & Related papers  (2022-06-06T18:07:41Z)
• Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning [125.8224674893018]
  Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Applying off-policy algorithms to offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. We propose Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven offline algorithm without explicit policy constraints.
  arXiv  Detail & Related papers  (2022-02-23T15:27:16Z)
• Safe Reinforcement Learning for Grid Voltage Control [0.0]
  Under voltage load shedding has been considered as a standard approach to recover the voltage stability of the electric power grid under emergency conditions. In this paper, we discuss a couple of novel safe RL approaches, namely constrained optimization approach and Barrier function-based approach.
  arXiv  Detail & Related papers  (2021-12-02T18:34:50Z)
• Instabilities of Offline RL with Pre-Trained Neural Representation [127.89397629569808]
  In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated. Recent theoretical advances have shown that such sample-efficient offline RL is indeed possible provided certain strong representational conditions hold. This work studies these issues from an empirical perspective to gauge how stable offline RL methods are.
  arXiv  Detail & Related papers  (2021-03-08T18:06:44Z)
• Enhancing the Spatio-temporal Observability of Grid-Edge Resources in Distribution Grids [5.815007821143811]
  Enhancing the convex-temporal observability of distributed energy resources (DERs) is crucial for achieving secure and efficient operations in distribution grids. This paper puts forth a joint recovery framework for residential loads by leveraging the complimentary strengths of heterogeneous types of measurements.
  arXiv  Detail & Related papers  (2021-02-15T19:12:54Z)
• Deep Reinforcement Learning for Electric Transmission Voltage Control [0.0]
  A subset of machine learning known as deep reinforcement learning (DRL) has recently shown promise in performing tasks typically performed by humans. This paper applies DRL to the transmission voltage control problem, presents open-source DRL environments for voltage control, and performs experiments at scale with systems up to 500 buses.
  arXiv  Detail & Related papers  (2020-06-11T18:35:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.