Stepping Out of the Shadows: Reinforcement Learning in Shadow Mode

• URL: http://arxiv.org/abs/2410.23419v1
• Date: Wed, 30 Oct 2024 19:52:52 GMT
• Title: Stepping Out of the Shadows: Reinforcement Learning in Shadow Mode
• Authors: Philipp Gassert, Matthias Althoff,
• Abstract summary: Reinforcement learning is not yet competitive for many cyber-physical systems. We train the reinforcement agent in a so-called shadow mode with the assistance of an existing conventional controller. In shadow mode, the agent relies on the controller to provide action samples and guidance towards favourable states to learn the task.
• Score: 8.017543518311196
• License:
• Abstract: Reinforcement learning (RL) is not yet competitive for many cyber-physical systems, such as robotics, process automation, and power systems, as training on a system with physical components cannot be accelerated, and simulation models do not exist or suffer from a large simulation-to-reality gap. During the long training time, expensive equipment cannot be used and might even be damaged due to inappropriate actions of the reinforcement learning agent. Our novel approach addresses exactly this problem: We train the reinforcement agent in a so-called shadow mode with the assistance of an existing conventional controller, which does not have to be trained and instantaneously performs reasonably well. In shadow mode, the agent relies on the controller to provide action samples and guidance towards favourable states to learn the task, while simultaneously estimating for which states the learned agent will receive a higher reward than the conventional controller. The RL agent will then control the system for these states and all other regions remain under the control of the existing controller. Over time, the RL agent will take over for an increasing amount of states, while leaving control to the baseline, where it cannot surpass its performance. Thus, we keep regret during training low and improve the performance compared to only using conventional controllers or reinforcement learning. We present and evaluate two mechanisms for deciding whether to use the RL agent or the conventional controller. The usefulness of our approach is demonstrated for a reach-avoid task, for which we are able to effectively train an agent, where standard approaches fail.

Related papers

• Learning to Fly in Seconds [7.259696592534715]
  We show how curriculum learning and a highly optimized simulator enhance sample complexity and lead to fast training times. Our framework enables Simulation-to-Reality (Sim2Real) transfer for direct control after only 18 seconds of training on a consumer-grade laptop.
  arXiv  Detail & Related papers  (2023-11-22T01:06:45Z)
• Reinforcement Learning for UAV control with Policy and Reward Shaping [0.7127008801193563]
  This study teaches an RL agent to control a drone using reward-shaping and policy-shaping techniques simultaneously. The results show that an agent trained simultaneously with both techniques obtains a lower reward than an agent trained using only a policy-based approach.
  arXiv  Detail & Related papers  (2022-12-06T14:46:13Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Skip Training for Multi-Agent Reinforcement Learning Controller for Industrial Wave Energy Converters [94.84709449845352]
  Recent Wave Energy Converters (WEC) are equipped with multiple legs and generators to maximize energy generation. Traditional controllers have shown limitations to capture complex wave patterns and the controllers must efficiently maximize the energy capture. This paper introduces a Multi-Agent Reinforcement Learning controller (MARL), which outperforms the traditionally used spring damper controller.
  arXiv  Detail & Related papers  (2022-09-13T00:20:31Z)
• Coach-assisted Multi-Agent Reinforcement Learning Framework for Unexpected Crashed Agents [120.91291581594773]
  We present a formal formulation of a cooperative multi-agent reinforcement learning system with unexpected crashes. We propose a coach-assisted multi-agent reinforcement learning framework, which introduces a virtual coach agent to adjust the crash rate during training. To the best of our knowledge, this work is the first to study the unexpected crashes in the multi-agent system.
  arXiv  Detail & Related papers  (2022-03-16T08:22:45Z)
• Zero-Shot Uncertainty-Aware Deployment of Simulation Trained Policies on Real-World Robots [17.710172337571617]
  Deep reinforcement learning (RL) agents tend to make errors when deployed in the real world due to mismatches between the training and execution environments. We propose a novel uncertainty-aware deployment strategy that combines the strengths of deep RL policies and traditional handcrafted controllers. We show promising results on two real-world continuous control tasks, where BCF outperforms both the standalone policy and controller.
  arXiv  Detail & Related papers  (2021-12-10T02:13:01Z)
• URLB: Unsupervised Reinforcement Learning Benchmark [82.36060735454647]
  We introduce the Unsupervised Reinforcement Learning Benchmark (URLB) URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. We provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods.
  arXiv  Detail & Related papers  (2021-10-28T15:07:01Z)
• Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training [71.7750435554693]
  We show that several state-of-the-art RL agents proposed for power system control are vulnerable to adversarial attacks. Specifically, we use an adversary Markov Decision Process to learn an attack policy, and demonstrate the potency of our attack. We propose to use adversarial training to increase the robustness of RL agent against attacks and avoid infeasible operational decisions.
  arXiv  Detail & Related papers  (2021-10-18T00:50:34Z)
• Regularizing Action Policies for Smooth Control with Reinforcement Learning [47.312768123967025]
  Conditioning for Action Policy Smoothness (CAPS) is an effective yet intuitive regularization on action policies. CAPS offers consistent improvement in the smoothness of the learned state-to-action mappings of neural network controllers. Tested on a real system, improvements in controller smoothness on a quadrotor drone resulted in an almost 80% reduction in power consumption.
  arXiv  Detail & Related papers  (2020-12-11T21:35:24Z)
• Robustifying Reinforcement Learning Agents via Action Space Adversarial Training [23.284452331353894]
  Adoption of machine learning (ML)-enabled cyber-physical systems (CPS) are becoming prevalent in various sectors of modern society. Recent studies in deep reinforcement learning (DRL) have demonstrated its benefits in a large variety of data-driven decisions and control applications. We show that a well-performing DRL agent that is initially susceptible to action space perturbations can be robustified against similar perturbations through adversarial training.
  arXiv  Detail & Related papers  (2020-07-14T16:50:02Z)
• Learning Force Control for Contact-rich Manipulation Tasks with Rigid Position-controlled Robots [9.815369993136512]
  We propose a learning-based force control framework combining RL techniques with traditional force control. Within said control scheme, we implemented two different conventional approaches to achieve force control with position-controlled robots. Finally, we developed a fail-safe mechanism for safely training an RL agent on manipulation tasks using a real rigid robot manipulator.
  arXiv  Detail & Related papers  (2020-03-02T01:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.