Robust Defense Against Extreme Grid Events Using Dual-Policy Reinforcement Learning Agents

• URL: http://arxiv.org/abs/2411.11180v1
• Date: Sun, 17 Nov 2024 21:30:48 GMT
• Title: Robust Defense Against Extreme Grid Events Using Dual-Policy Reinforcement Learning Agents
• Authors: Benjamin M. Peter, Mert Korkali,
• Abstract summary: Reinforcement learning (RL) agents are powerful tools for managing power grids. They use large amounts of data to inform their actions and receive rewards as feedback to learn favorable responses for the system. This ability is especially valuable in decarbonizing power networks, where the demand for RL agents is increasing.
• Score: 0.0
• License:
• Abstract: Reinforcement learning (RL) agents are powerful tools for managing power grids. They use large amounts of data to inform their actions and receive rewards or penalties as feedback to learn favorable responses for the system. Once trained, these agents can efficiently make decisions that would be too computationally complex for a human operator. This ability is especially valuable in decarbonizing power networks, where the demand for RL agents is increasing. These agents are well suited to control grid actions since the action space is constantly growing due to uncertainties in renewable generation, microgrid integration, and cybersecurity threats. To assess the efficacy of RL agents in response to an adverse grid event, we use the Grid2Op platform for agent training. We employ a proximal policy optimization (PPO) algorithm in conjunction with graph neural networks (GNNs). By simulating agents' responses to grid events, we assess their performance in avoiding grid failure for as long as possible. The performance of an agent is expressed concisely through its reward function, which helps the agent learn the most optimal ways to reconfigure a grid's topology amidst certain events. To model multi-actor scenarios that threaten modern power networks, particularly those resulting from cyberattacks, we integrate an opponent that acts iteratively against a given agent. This interplay between the RL agent and opponent is utilized in N-k contingency screening, providing a novel alternative to the traditional security assessment.

Related papers

• Multi-Objective Reinforcement Learning for Automated Resilient Cyber Defence [0.0]
  Cyber-attacks pose a security threat to military command and control networks, Intelligence, Surveillance, and Reconnaissance (ISR) systems, and civilian critical national infrastructure. The use of artificial intelligence and autonomous agents in these attacks increases the scale, range, and complexity of this threat and the subsequent disruption they cause. Autonomous Cyber Defence (ACD) agents aim to mitigate this threat by responding at machine speed and at the scale required to address the problem.
  arXiv  Detail & Related papers  (2024-11-26T16:51:52Z)
• Imitation Learning for Intra-Day Power Grid Operation through Topology Actions [0.24578723416255752]
  We study the performance of imitation learning for day-ahead power grid operation through topology actions. We train a fully-connected neural network (FCNN) on expert state-action pairs and evaluate it in two ways. As a power system agent, the FCNN performs only slightly worse than expert agents.
  arXiv  Detail & Related papers  (2024-07-29T10:34:19Z)
• Toward Optimal LLM Alignments Using Two-Player Games [86.39338084862324]
  In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.
  arXiv  Detail & Related papers  (2024-06-16T15:24:50Z)
• Towards Autonomous Cyber Operation Agents: Exploring the Red Case [3.805031560408777]
  Reinforcement and deep reinforcement learning (RL/DRL) have been applied to develop autonomous agents for cyber network operations (CyOps) The training environment must simulate CyOps with high fidelity, which the agent aims to learn and accomplish. A good simulator is hard to achieve due to the extreme complexity of the cyber environment.
  arXiv  Detail & Related papers  (2023-09-05T13:56:31Z)
• Graph Neural Networks for Decentralized Multi-Agent Perimeter Defense [111.9039128130633]
  We develop an imitation learning framework that learns a mapping from defenders' local perceptions and their communication graph to their actions. We run perimeter defense games in scenarios with different team sizes and configurations to demonstrate the performance of the learned network.
  arXiv  Detail & Related papers  (2023-01-23T19:35:59Z)
• Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading [0.19116784879310028]
  This paper describes how domain knowledge of power system operators can be integrated into reinforcement learning frameworks. A curriculum-based approach with reward tuning is incorporated into the training procedure by modifying the environment. A parallel training approach on multiple scenarios is employed to avoid biasing the agent to a few scenarios and make it robust to the natural variability in grid operations.
  arXiv  Detail & Related papers  (2021-12-18T20:32:05Z)
• Autonomous Attack Mitigation for Industrial Control Systems [25.894883701063055]
  Defending computer networks from cyber attack requires timely responses to alerts and threat intelligence. We present a deep reinforcement learning approach to autonomous response and recovery in large industrial control networks.
  arXiv  Detail & Related papers  (2021-11-03T18:08:06Z)
• Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training [71.7750435554693]
  We show that several state-of-the-art RL agents proposed for power system control are vulnerable to adversarial attacks. Specifically, we use an adversary Markov Decision Process to learn an attack policy, and demonstrate the potency of our attack. We propose to use adversarial training to increase the robustness of RL agent against attacks and avoid infeasible operational decisions.
  arXiv  Detail & Related papers  (2021-10-18T00:50:34Z)
• Explore and Control with Adversarial Surprise [78.41972292110967]
  Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-specified rewards. We propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences. We show that our method leads to the emergence of complex skills by exhibiting clear phase transitions.
  arXiv  Detail & Related papers  (2021-07-12T17:58:40Z)
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble [135.6115462399788]
  Deep offline reinforcement learning has made it possible to train strong robotic agents from offline datasets. State-action distribution shift may lead to severe bootstrap error during fine-tuning. We propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples.
  arXiv  Detail & Related papers  (2021-07-01T16:26:54Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.