Globally Stable Neural Imitation Policies

• URL: http://arxiv.org/abs/2403.04118v2
• Date: Mon, 2 Sep 2024 18:03:26 GMT
• Title: Globally Stable Neural Imitation Policies
• Authors: Amin Abyaneh, Mariana Sosa Guzmán, Hsiu-Chin Lin,
• Abstract summary: We introduce the Stable Neural Dynamical System (SNDS), an imitation learning regime which produces a policy with formal stability guarantees. We deploy a neural policy architecture that facilitates the representation of stability based on Lyapunov theorem. We successfully deploy the trained policies on a real-world manipulator arm.
• Score: 3.8772936189143445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation learning presents an effective approach to alleviate the resource-intensive and time-consuming nature of policy learning from scratch in the solution space. Even though the resulting policy can mimic expert demonstrations reliably, it often lacks predictability in unexplored regions of the state-space, giving rise to significant safety concerns in the face of perturbations. To address these challenges, we introduce the Stable Neural Dynamical System (SNDS), an imitation learning regime which produces a policy with formal stability guarantees. We deploy a neural policy architecture that facilitates the representation of stability based on Lyapunov theorem, and jointly train the policy and its corresponding Lyapunov candidate to ensure global stability. We validate our approach by conducting extensive experiments in simulation and successfully deploying the trained policies on a real-world manipulator arm. The experimental results demonstrate that our method overcomes the instability, accuracy, and computational intensity problems associated with previous imitation learning methods, making our method a promising solution for stable policy learning in complex planning scenarios.

Related papers

• COIN: Chance-Constrained Imitation Learning for Uncertainty-aware Adaptive Resource Oversubscription Policy [37.034543365623286]
  We address the challenge of learning safe and robust decision policies in presence of uncertainty. Traditional supervised prediction or forecasting models are ineffective in learning adaptive policies. Online optimization or reinforcement learning is difficult to deploy on real systems.
  arXiv  Detail & Related papers  (2024-01-13T11:43:25Z)
• Probabilistic Reach-Avoid for Bayesian Neural Networks [71.67052234622781]
  We show that an optimal synthesis algorithm can provide more than a four-fold increase in the number of certifiable states. The algorithm is able to provide more than a three-fold increase in the average guaranteed reach-avoid probability.
  arXiv  Detail & Related papers  (2023-10-03T10:52:21Z)
• Learning Provably Stabilizing Neural Controllers for Discrete-Time Stochastic Systems [18.349820472823055]
  We introduce the notion of stabilizing ranking supermartingales (sRSMs) We show that our learning procedure can successfully learn provably stabilizing policies in practice.
  arXiv  Detail & Related papers  (2022-10-11T09:55:07Z)
• KCRL: Krasovskii-Constrained Reinforcement Learning with Guaranteed Stability in Nonlinear Dynamical Systems [66.9461097311667]
  We propose a model-based reinforcement learning framework with formal stability guarantees. The proposed method learns the system dynamics up to a confidence interval using feature representation. We show that KCRL is guaranteed to learn a stabilizing policy in a finite number of interactions with the underlying unknown system.
  arXiv  Detail & Related papers  (2022-06-03T17:27:04Z)
• Learning Stabilizing Policies in Stochastic Control Systems [20.045860624444494]
  We study the effectiveness of jointly learning a policy together with a martingale certificate that proves its stability using a single learning algorithm. Our results suggest that some form of pre-training of the policy is required for the joint optimization to repair and verify the policy successfully.
  arXiv  Detail & Related papers  (2022-05-24T11:38:22Z)
• On Imitation Learning of Linear Control Policies: Enforcing Stability and Robustness Constraints via LMI Conditions [3.296303220677533]
  We formulate the imitation learning of linear policies as a constrained optimization problem. We show that one can guarantee the closed-loop stability and robustness by posing linear matrix inequality (LMI) constraints on the fitted policy.
  arXiv  Detail & Related papers  (2021-03-24T02:43:03Z)
• Closing the Closed-Loop Distribution Shift in Safe Imitation Learning [80.05727171757454]
  We treat safe optimization-based control strategies as experts in an imitation learning problem. We train a learned policy that can be cheaply evaluated at run-time and that provably satisfies the same safety guarantees as the expert.
  arXiv  Detail & Related papers  (2021-02-18T05:11:41Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Improving Robustness via Risk Averse Distributional Reinforcement Learning [13.467017642143581]
  Robustness is critical when the policies are trained in simulations instead of real world environment. We propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation.
  arXiv  Detail & Related papers  (2020-05-01T20:03:10Z)
• Deep Reinforcement Learning with Robust and Smooth Policy [90.78795857181727]
  We propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- textbfSmooth textbfRegularized textbfReinforcement textbfLearning ($textbfSR2textbfL$), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy.
  arXiv  Detail & Related papers  (2020-03-21T00:10:29Z)
• Stable Policy Optimization via Off-Policy Divergence Regularization [50.98542111236381]
  Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL) We propose a new algorithm which stabilizes the policy improvement through a proximity term that constrains the discounted state-action visitation distribution induced by consecutive policies to be close to one another. Our proposed method can have a beneficial effect on stability and improve final performance in benchmark high-dimensional control tasks.
  arXiv  Detail & Related papers  (2020-03-09T13:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.