Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies

• URL: http://arxiv.org/abs/2405.01198v1
• Date: Thu, 2 May 2024 11:40:15 GMT
• Title: Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies
• Authors: Finn Rietz, Erik Schaffernicht, Stefan Heinrich, Johannes A. Stork,
• Abstract summary: Reinforcement learning policies are typically represented by black-box neural networks. We propose constrained normalizing flow policies as interpretable and safe-by-construction policy models.
• Score: 5.6872893893453105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning policies are typically represented by black-box neural networks, which are non-interpretable and not well-suited for safety-critical domains. To address both of these issues, we propose constrained normalizing flow policies as interpretable and safe-by-construction policy models. We achieve safety for reinforcement learning problems with instantaneous safety constraints, for which we can exploit domain knowledge by analytically constructing a normalizing flow that ensures constraint satisfaction. The normalizing flow corresponds to an interpretable sequence of transformations on action samples, each ensuring alignment with respect to a particular constraint. Our experiments reveal benefits beyond interpretability in an easier learning objective and maintained constraint satisfaction throughout the entire learning process. Our approach leverages constraints over reward engineering while offering enhanced interpretability, safety, and direct means of providing domain knowledge to the agent without relying on complex reward functions.

Related papers

• Feasibility Consistent Representation Learning for Safe Reinforcement Learning [25.258227763316228]
  We introduce a novel framework named Feasibility Consistent Safe Reinforcement Learning (FCSRL) This framework combines representation learning with feasibility-oriented objectives to identify and extract safety-related information from the raw state for safe RL. Our method is capable of learning a better safety-aware embedding and achieving superior performance than previous representation learning baselines.
  arXiv  Detail & Related papers  (2024-05-20T01:37:21Z)
• A Survey of Constraint Formulations in Safe Reinforcement Learning [15.593999581562203]
  Safety is critical when applying reinforcement learning to real-world problems. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult.
  arXiv  Detail & Related papers  (2024-02-03T04:40:31Z)
• Safety-Constrained Policy Transfer with Successor Features [19.754549649781644]
  We propose a Constrained Markov Decision Process (CMDP) formulation that enables the transfer of policies and adherence to safety constraints. Our approach relies on a novel extension of generalized policy improvement to constrained settings via a Lagrangian formulation. Our experiments in simulated domains show that our approach is effective; it visits unsafe states less frequently and outperforms alternative state-of-the-art methods when taking safety constraints into account.
  arXiv  Detail & Related papers  (2022-11-10T06:06:36Z)
• Safe Exploration Method for Reinforcement Learning under Existence of Disturbance [1.1470070927586016]
  We deal with a safe exploration problem in reinforcement learning under the existence of disturbance. We propose a safe exploration method that uses partial prior knowledge of a controlled object and disturbance. We illustrate the validity and effectiveness of the proposed method through numerical simulations of an inverted pendulum and a four-bar parallel link robot manipulator.
  arXiv  Detail & Related papers  (2022-09-30T13:00:33Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates [63.58053355357644]
  We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks. We show theoretically that having access to a specific set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance.
  arXiv  Detail & Related papers  (2021-12-30T12:20:46Z)
• Learning to Be Cautious [71.9871661858886]
  A key challenge in the field of reinforcement learning is to develop agents that behave cautiously in novel situations. We present a sequence of tasks where cautious behavior becomes increasingly non-obvious, as well as an algorithm to demonstrate that it is possible for a system to emphlearn to be cautious.
  arXiv  Detail & Related papers  (2021-10-29T16:52:45Z)
• Closing the Closed-Loop Distribution Shift in Safe Imitation Learning [80.05727171757454]
  We treat safe optimization-based control strategies as experts in an imitation learning problem. We train a learned policy that can be cheaply evaluated at run-time and that provably satisfies the same safety guarantees as the expert.
  arXiv  Detail & Related papers  (2021-02-18T05:11:41Z)
• Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients [54.98496284653234]
  We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions. We solve this problem by introducing a regularizer based on the mutual information between the sensitive state and the actions. We develop a model-based estimator for optimization of privacy-constrained policies.
  arXiv  Detail & Related papers  (2020-12-30T03:22:35Z)
• Regularized Inverse Reinforcement Learning [49.78352058771138]
  Inverse Reinforcement Learning (IRL) aims to facilitate a learner's ability to imitate expert behavior. Regularized IRL applies strongly convex regularizers to the learner's policy. We propose tractable solutions, and practical methods to obtain them, for regularized IRL.
  arXiv  Detail & Related papers  (2020-10-07T23:38:47Z)
• Constrained Markov Decision Processes via Backward Value Functions [43.649330976089004]
  We model the problem of learning with constraints as a Constrained Markov Decision Process. A key contribution of our approach is to translate cumulative cost constraints into state-based constraints. We provide theoretical guarantees under which the agent converges while ensuring safety over the course of training.
  arXiv  Detail & Related papers  (2020-08-26T20:56:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.