Reinforcement Learning of Structured Control for Linear Systems with Unknown State Matrix

• URL: http://arxiv.org/abs/2011.01128v1
• Date: Mon, 2 Nov 2020 17:04:34 GMT
• Title: Reinforcement Learning of Structured Control for Linear Systems with Unknown State Matrix
• Authors: Sayak Mukherjee, Thanh Long Vu
• Abstract summary: We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees. A special control structure enabled by this RL framework is distributed learning control which is necessary for many large-scale cyber-physical systems.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper delves into designing stabilizing feedback control gains for continuous linear systems with unknown state matrix, in which the control is subject to a general structural constraint. We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees in order to design these structured gains using the trajectory measurements of states and controls. We first formulate a model-based framework using dynamic programming (DP) to embed the structural constraint to the Linear Quadratic Regulator (LQR) gain computation in the continuous-time setting. Subsequently, we transform this LQR formulation into a policy iteration RL algorithm that can alleviate the requirement of known state matrix in conjunction with maintaining the feedback gain structure. Theoretical guarantees are provided for stability and convergence of the structured RL (SRL) algorithm. The introduced RL framework is general and can be applied to any control structure. A special control structure enabled by this RL framework is distributed learning control which is necessary for many large-scale cyber-physical systems. As such, we validate our theoretical results with numerical simulations on a multi-agent networked linear time-invariant (LTI) dynamic system.

Related papers

• Stable Modular Control via Contraction Theory for Reinforcement Learning [8.742125999252366]
  We propose a novel way to integrate control techniques with reinforcement learning (RL) for stability, robustness, and generalization. We realize such modularity via signal composition and dynamic decomposition.
  arXiv  Detail & Related papers  (2023-11-07T02:41:02Z)
• Sparsity in Partially Controllable Linear Systems [56.142264865866636]
  We study partially controllable linear dynamical systems specified by an underlying sparsity pattern. Our results characterize those state variables which are irrelevant for optimal control.
  arXiv  Detail & Related papers  (2021-10-12T16:41:47Z)
• Stable Online Control of Linear Time-Varying Systems [49.41696101740271]
  COCO-LQ is an efficient online control algorithm that guarantees input-to-state stability for a large class of LTV systems. We empirically demonstrate the performance of COCO-LQ in both synthetic experiments and a power system frequency control example.
  arXiv  Detail & Related papers  (2021-04-29T06:18:49Z)
• RL-Controller: a reinforcement learning framework for active structural control [0.0]
  We present a novel RL-based approach for designing active controllers by introducing RL-Controller, a flexible and scalable simulation environment. We show that the proposed framework is easily trainable for a five story benchmark building with 65% reductions on average in inter story drifts. In a comparative study with LQG active control method, we demonstrate that the proposed model-free algorithm learns more optimal actuator forcing strategies.
  arXiv  Detail & Related papers  (2021-03-13T04:42:13Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Imposing Robust Structured Control Constraint on Reinforcement Learning of Linear Quadratic Regulator [0.0]
  This paper presents a design for any generic structure, paving the way for distributed learning control. The ideas from reinforcement learning (RL) in conjunction with control-theoretic sufficient stability and performance guarantees are used to develop the methodology. We validate our theoretical results with a simulation on a multi-agent network with 6 agents.
  arXiv  Detail & Related papers  (2020-11-12T00:31:39Z)
• Reduced-Dimensional Reinforcement Learning Control using Singular Perturbation Approximations [9.136645265350284]
  We present a set of model-free, reduced-dimensional reinforcement learning based optimal control designs for linear time-invariant singularly perturbed (SP) systems. We first present a state-feedback and output-feedback based RL control design for a generic SP system with unknown state and input matrices. We extend both designs to clustered multi-agent consensus networks, where the SP property reflects through clustering.
  arXiv  Detail & Related papers  (2020-04-29T22:15:54Z)
• Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting [91.43582419264763]
  We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty. LqgOpt efficiently explores the system dynamics, estimates the model parameters up to their confidence interval, and deploys the controller of the most optimistic model.
  arXiv  Detail & Related papers  (2020-03-12T19:56:38Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)
• Certified Reinforcement Learning with Logic Guidance [78.2286146954051]
  We propose a model-free RL algorithm that enables the use of Linear Temporal Logic (LTL) to formulate a goal for unknown continuous-state/action Markov Decision Processes (MDPs) The algorithm is guaranteed to synthesise a control policy whose traces satisfy the specification with maximal probability.
  arXiv  Detail & Related papers  (2019-02-02T20:09:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.