Steady-State Error Compensation in Reference Tracking and Disturbance Rejection Problems for Reinforcement Learning-Based Control

• URL: http://arxiv.org/abs/2201.13331v1
• Date: Mon, 31 Jan 2022 16:29:19 GMT
• Title: Steady-State Error Compensation in Reference Tracking and Disturbance Rejection Problems for Reinforcement Learning-Based Control
• Authors: Daniel Weber, Maximilian Schenke and Oliver Wallscheid
• Abstract summary: Reinforcement learning (RL) is a promising, upcoming topic in automatic control applications. Initiative action state augmentation (IASA) for actor-critic-based RL controllers is introduced. This augmentation does not require any expert knowledge, leaving the approach model free.
• Score: 0.9023847175654602
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) is a promising, upcoming topic in automatic control applications. Where classical control approaches require a priori system knowledge, data-driven control approaches like RL allow a model-free controller design procedure, rendering them emergent techniques for systems with changing plant structures and varying parameters. While it was already shown in various applications that the transient control behavior for complex systems can be sufficiently handled by RL, the challenge of non-vanishing steady-state control errors remains, which arises from the usage of control policy approximations and finite training times. To overcome this issue, an integral action state augmentation (IASA) for actor-critic-based RL controllers is introduced that mimics an integrating feedback, which is inspired by the delta-input formulation within model predictive control. This augmentation does not require any expert knowledge, leaving the approach model free. As a result, the RL controller learns how to suppress steady-state control deviations much more effectively. Two exemplary applications from the domain of electrical energy engineering validate the benefit of the developed method both for reference tracking and disturbance rejection. In comparison to a standard deep deterministic policy gradient (DDPG) setup, the suggested IASA extension allows to reduce the steady-state error by up to 52 $\%$ within the considered validation scenarios.

Related papers

• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
  In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
  arXiv  Detail & Related papers  (2024-04-05T17:58:37Z)
• A Safe Reinforcement Learning Algorithm for Supervisory Control of Power Plants [7.1771300511732585]
  Model-free Reinforcement learning (RL) has emerged as a promising solution for control tasks. We propose a chance-constrained RL algorithm based on Proximal Policy Optimization for supervisory control. Our approach achieves the smallest distance of violation and violation rate in a load-follow maneuver for an advanced Nuclear Power Plant design.
  arXiv  Detail & Related papers  (2024-01-23T17:52:49Z)
• Controllability-Constrained Deep Network Models for Enhanced Control of Dynamical Systems [4.948174943314265]
  Control of a dynamical system without the knowledge of dynamics is an important and challenging task. Modern machine learning approaches, such as deep neural networks (DNNs), allow for the estimation of a dynamics model from control inputs and corresponding state observation outputs. We propose a control-theoretical method that explicitly enhances models estimated from data with controllability.
  arXiv  Detail & Related papers  (2023-11-11T00:04:26Z)
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
  We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
  arXiv  Detail & Related papers  (2023-04-20T17:11:05Z)
• Sparsity in Partially Controllable Linear Systems [56.142264865866636]
  We study partially controllable linear dynamical systems specified by an underlying sparsity pattern. Our results characterize those state variables which are irrelevant for optimal control.
  arXiv  Detail & Related papers  (2021-10-12T16:41:47Z)
• Finite-time System Identification and Adaptive Control in Autoregressive Exogenous Systems [79.67879934935661]
  We study the problem of system identification and adaptive control of unknown ARX systems. We provide finite-time learning guarantees for the ARX systems under both open-loop and closed-loop data collection.
  arXiv  Detail & Related papers  (2021-08-26T18:00:00Z)
• Regret Analysis of Learning-Based MPC with Partially-Unknown Cost Function [5.601217969637838]
  exploration/exploitation trade-off is an inherent challenge in data-driven and adaptive control. We propose the use of a finitehorizon oracle controller with perfect knowledge of all system parameters as a reference for optimal control actions. We develop learning-based policies that we prove achieve low regret with respect to this oracle finite-horizon controller.
  arXiv  Detail & Related papers  (2021-08-04T22:43:51Z)
• Regret-optimal Estimation and Control [52.28457815067461]
  We show that the regret-optimal estimator and regret-optimal controller can be derived in state-space form. We propose regret-optimal analogs of Model-Predictive Control (MPC) and the Extended KalmanFilter (EKF) for systems with nonlinear dynamics.
  arXiv  Detail & Related papers  (2021-06-22T23:14:21Z)
• Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty [22.276925045008788]
  emphglsrpl offers a formulation to improve existing controllers with reinforcement learning (RL) We show superior performance of our approach on a contact-rich peg-insertion task under position and orientation uncertainty.
  arXiv  Detail & Related papers  (2021-06-08T13:06:35Z)
• Reinforcement Learning for Thermostatically Controlled Loads Control using Modelica and Python [0.0]
  The project aims to investigate and assess opportunities for applying reinforcement learning (RL) for power system control. As a proof of concept (PoC), voltage control of thermostatically controlled loads (TCLs) for power consumption was developed using Modelica-based pipeline. The paper shows the influence of Q-learning parameters, including discretization of state-action space, on the controller performance.
  arXiv  Detail & Related papers  (2020-05-09T13:35:49Z)
• Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting [91.43582419264763]
  We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty. LqgOpt efficiently explores the system dynamics, estimates the model parameters up to their confidence interval, and deploys the controller of the most optimistic model.
  arXiv  Detail & Related papers  (2020-03-12T19:56:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.