A modular framework for stabilizing deep reinforcement learning control

• URL: http://arxiv.org/abs/2304.03422v1
• Date: Fri, 7 Apr 2023 00:09:17 GMT
• Title: A modular framework for stabilizing deep reinforcement learning control
• Authors: Nathan P. Lawrence, Philip D. Loewen, Shuyuan Wang, Michael G. Forbes, R. Bhushan Gopaluni
• Abstract summary: We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees. Recent advances in behavioral systems allow us to construct a data-driven internal model. This enables an alternative realization of the Youla-Kucera parameterization based entirely on input-output exploration data.
• Score: 3.3598755777055374
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees provided by using the Youla-Kucera parameterization to define the search domain. Recent advances in behavioral systems allow us to construct a data-driven internal model; this enables an alternative realization of the Youla-Kucera parameterization based entirely on input-output exploration data. Using a neural network to express a parameterized set of nonlinear stable operators enables seamless integration with standard deep learning libraries. We demonstrate the approach on a realistic simulation of a two-tank system.

Related papers

• Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset [98.52916361979503]
  We introduce a novel learning approach that automatically models and adapts to non-stationarity. We show empirically that our approach performs well in non-stationary supervised and off-policy reinforcement learning settings.
  arXiv  Detail & Related papers  (2024-11-06T16:32:40Z)
• Continual Model-based Reinforcement Learning for Data Efficient Wireless Network Optimisation [73.04087903322237]
  We formulate throughput optimisation as Continual Reinforcement Learning of control policies. Simulation results suggest that the proposed system is able to shorten the end-to-end deployment lead-time by two-fold.
  arXiv  Detail & Related papers  (2024-04-30T11:23:31Z)
• Data-driven Nonlinear Model Reduction using Koopman Theory: Integrated Control Form and NMPC Case Study [56.283944756315066]
  We propose generic model structures combining delay-coordinate encoding of measurements and full-state decoding to integrate reduced Koopman modeling and state estimation. A case study demonstrates that our approach provides accurate control models and enables real-time capable nonlinear model predictive control of a high-purity cryogenic distillation column.
  arXiv  Detail & Related papers  (2024-01-09T11:54:54Z)
• Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior [2.4641488282873225]
  We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees. Recent advances in behavioral systems allow us to construct a data-driven internal model. We analyze the stability of such data-driven models in the presence of noise.
  arXiv  Detail & Related papers  (2023-10-21T19:32:11Z)
• Learning Over Contracting and Lipschitz Closed-Loops for Partially-Observed Nonlinear Systems (Extended Version) [1.2430809884830318]
  This paper presents a policy parameterization for learning-based control on nonlinear, partially-observed dynamical systems. We prove that the resulting Youla-REN parameterization automatically satisfies stability (contraction) and user-tunable robustness (Lipschitz) conditions. We find that the Youla-REN performs similarly to existing learning-based and optimal control methods while also ensuring stability and exhibiting improved robustness to adversarial disturbances.
  arXiv  Detail & Related papers  (2023-04-12T23:55:56Z)
• On the Forward Invariance of Neural ODEs [92.07281135902922]
  We propose a new method to ensure neural ordinary differential equations (ODEs) satisfy output specifications. Our approach uses a class of control barrier functions to transform output specifications into constraints on the parameters and inputs of the learning system.
  arXiv  Detail & Related papers  (2022-10-10T15:18:28Z)
• Neural Lyapunov Differentiable Predictive Control [2.042924346801313]
  We present a learning-based predictive control methodology using the differentiable programming framework with probabilistic Lyapunov-based stability guarantees. In conjunction, our approach jointly learns a Lyapunov function that certifies the regions of state-space with stable dynamics.
  arXiv  Detail & Related papers  (2022-05-22T03:52:27Z)
• Youla-REN: Learning Nonlinear Feedback Policies with Robust Stability Guarantees [5.71097144710995]
  This paper presents a parameterization of nonlinear controllers for uncertain systems building on a recently developed neural network architecture. The proposed framework has "built-in" guarantees of stability, i.e., all policies in the search space result in a contracting (globally exponentially stable) closed-loop system.
  arXiv  Detail & Related papers  (2021-12-02T13:52:37Z)
• Learning Stable Koopman Embeddings [9.239657838690228]
  We present a new data-driven method for learning stable models of nonlinear systems. We prove that every discrete-time nonlinear contracting model can be learnt in our framework.
  arXiv  Detail & Related papers  (2021-10-13T05:44:13Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Learning Control Barrier Functions from Expert Demonstrations [69.23675822701357]
  We propose a learning based approach to safe controller synthesis based on control barrier functions (CBFs) We analyze an optimization-based approach to learning a CBF that enjoys provable safety guarantees under suitable Lipschitz assumptions on the underlying dynamical system. To the best of our knowledge, these are the first results that learn provably safe control barrier functions from data.
  arXiv  Detail & Related papers  (2020-04-07T12:29:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.