Model-Based Reinforcement Learning via Stochastic Hybrid Models

• URL: http://arxiv.org/abs/2111.06211v3
• Date: Tue, 20 Jun 2023 05:10:37 GMT
• Title: Model-Based Reinforcement Learning via Stochastic Hybrid Models
• Authors: Hany Abdulsamad and Jan Peters
• Abstract summary: This paper adopts a hybrid-system view of nonlinear modeling and control. We consider a sequence modeling paradigm that captures the temporal structure of the data. We show that these time-series models naturally admit a closed-loop extension that we use to extract local feedback controllers.
• Score: 39.83837705993256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimal control of general nonlinear systems is a central challenge in automation. Enabled by powerful function approximators, data-driven approaches to control have recently successfully tackled challenging applications. However, such methods often obscure the structure of dynamics and control behind black-box over-parameterized representations, thus limiting our ability to understand closed-loop behavior. This paper adopts a hybrid-system view of nonlinear modeling and control that lends an explicit hierarchical structure to the problem and breaks down complex dynamics into simpler localized units. We consider a sequence modeling paradigm that captures the temporal structure of the data and derive an expectation-maximization (EM) algorithm that automatically decomposes nonlinear dynamics into stochastic piecewise affine models with nonlinear transition boundaries. Furthermore, we show that these time-series models naturally admit a closed-loop extension that we use to extract local polynomial feedback controllers from nonlinear experts via behavioral cloning. Finally, we introduce a novel hybrid relative entropy policy search (Hb-REPS) technique that incorporates the hierarchical nature of hybrid models and optimizes a set of time-invariant piecewise feedback controllers derived from a piecewise polynomial approximation of a global state-value function.

Related papers

• Random Features Approximation for Control-Affine Systems [6.067043299145924]
  We propose two novel classes of nonlinear feature representations which capture control affine structure. Our methods make use of random features (RF) approximations, inheriting the expressiveness of kernel methods at a lower computational cost.
  arXiv  Detail & Related papers  (2024-06-10T17:54:57Z)
• Data-driven Nonlinear Model Reduction using Koopman Theory: Integrated Control Form and NMPC Case Study [56.283944756315066]
  We propose generic model structures combining delay-coordinate encoding of measurements and full-state decoding to integrate reduced Koopman modeling and state estimation. A case study demonstrates that our approach provides accurate control models and enables real-time capable nonlinear model predictive control of a high-purity cryogenic distillation column.
  arXiv  Detail & Related papers  (2024-01-09T11:54:54Z)
• Neural Abstractions [72.42530499990028]
  We present a novel method for the safety verification of nonlinear dynamical models that uses neural networks to represent abstractions of their dynamics. We demonstrate that our approach performs comparably to the mature tool Flow* on existing benchmark nonlinear models.
  arXiv  Detail & Related papers  (2023-01-27T12:38:09Z)
• Formal Controller Synthesis for Markov Jump Linear Systems with Uncertain Dynamics [64.72260320446158]
  We propose a method for synthesising controllers for Markov jump linear systems. Our method is based on a finite-state abstraction that captures both the discrete (mode-jumping) and continuous (stochastic linear) behaviour of the MJLS. We apply our method to multiple realistic benchmark problems, in particular, a temperature control and an aerial vehicle delivery problem.
  arXiv  Detail & Related papers  (2022-12-01T17:36:30Z)
• Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach [0.0]
  We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems. The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning (in particular, non-linear manifold learning (DMs)) We exploit the Equation-free approach to perform numerical bifurcation analysis of the emergent dynamics. We design data-driven embedded wash-out controllers that drive the agent-based simulators to their intrinsic, imprecisely known, emergent open-loop unstable steady-states.
  arXiv  Detail & Related papers  (2022-07-12T18:16:22Z)
• Nonlinear proper orthogonal decomposition for convection-dominated flows [0.0]
  We propose an end-to-end Galerkin-free model combining autoencoders with long short-term memory networks for dynamics. Our approach not only improves the accuracy, but also significantly reduces the computational cost of training and testing.
  arXiv  Detail & Related papers  (2021-10-15T18:05:34Z)
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization [60.73540999409032]
  We show that expressive autoregressive dynamics models generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We also show that autoregressive dynamics models are useful for offline policy optimization by serving as a way to enrich the replay buffer.
  arXiv  Detail & Related papers  (2021-04-28T16:48:44Z)
• Combining Gaussian processes and polynomial chaos expansions for stochastic nonlinear model predictive control [0.0]
  We introduce a new algorithm to explicitly consider time-invariant uncertainties in optimal control problems. The main novelty in this paper is to use this combination in an efficient fashion to obtain mean and variance estimates of nonlinear transformations. It is shown how to formulate both chance-constraints and a probabilistic objective for the optimal control problem.
  arXiv  Detail & Related papers  (2021-03-09T14:25:08Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation [39.83837705993256]
  Current trends in reinforcement learning (RL) focus on complex representations of dynamics and policies. We take inspiration from the control community and apply the principles of hybrid switching systems in order to break down complex dynamics into simpler components.
  arXiv  Detail & Related papers  (2020-05-04T12:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.