System III: Learning with Domain Knowledge for Safety Constraints

• URL: http://arxiv.org/abs/2304.11593v1
• Date: Sun, 23 Apr 2023 09:44:41 GMT
• Title: System III: Learning with Domain Knowledge for Safety Constraints
• Abstract summary: This paper proposes a novel framework for incorporating domain knowledge to help guide safe exploration and boost sample efficiency. In our approach, called $textitSystem III$, we represent domain expert knowledge of safety in form of first-order logic. We evaluate the effectiveness of the proposed method on OpenAI's Gym and Safety-Gym environments.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning agents naturally learn from extensive exploration. Exploration is costly and can be unsafe in $\textit{safety-critical}$ domains. This paper proposes a novel framework for incorporating domain knowledge to help guide safe exploration and boost sample efficiency. Previous approaches impose constraints, such as regularisation parameters in neural networks, that rely on large sample sets and often are not suitable for safety-critical domains where agents should almost always avoid unsafe actions. In our approach, called $\textit{System III}$, which is inspired by psychologists' notions of the brain's $\textit{System I}$ and $\textit{System II}$, we represent domain expert knowledge of safety in form of first-order logic. We evaluate the satisfaction of these constraints via p-norms in state vector space. In our formulation, constraints are analogous to hazards, objects, and regions of state that have to be avoided during exploration. We evaluated the effectiveness of the proposed method on OpenAI's Gym and Safety-Gym environments. In all tasks, including classic Control and Safety Games, we show that our approach results in safer exploration and sample efficiency.

Related papers

• Saffron-1: Safety Inference Scaling [69.61130284742353]
  SAFFRON is a novel inference scaling paradigm tailored explicitly for safety assurance.<n>Central to our approach is the introduction of a multifurcation reward model (MRM) that significantly reduces the required number of reward model evaluations.<n>We publicly release our trained multifurcation reward model (Saffron-1) and the accompanying token-level safety reward dataset (Safety4M)
  arXiv  Detail & Related papers  (2025-06-06T18:05:45Z)
• Safe Guaranteed Exploration for Non-linear Systems [44.2908666969021]
  We propose a novel safe guaranteed exploration framework using optimal control, which achieves first-of-its-kind results. Based on this framework we propose an efficient algorithm, SageMPC, SAfe Guaranteed Exploration using Model Predictive Control. We demonstrate safe efficient exploration in challenging unknown environments using SageMPC with a car model.
  arXiv  Detail & Related papers  (2024-02-09T17:26:26Z)
• Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version) [1.279257604152629]
  Safe exploration aims at addressing the limitations of Reinforcement Learning (RL) in safety-critical scenarios. Several methods exist to incorporate external knowledge or to use sensor data to limit the exploration of unsafe states. In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement.
  arXiv  Detail & Related papers  (2023-07-10T22:28:33Z)
• Approximate Shielding of Atari Agents for Safe Exploration [83.55437924143615]
  We propose a principled algorithm for safe exploration based on the concept of shielding. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations.
  arXiv  Detail & Related papers  (2023-04-21T16:19:54Z)
• Safe Deep Reinforcement Learning by Verifying Task-Level Properties [84.64203221849648]
  Cost functions are commonly employed in Safe Deep Reinforcement Learning (DRL) The cost is typically encoded as an indicator function due to the difficulty of quantifying the risk of policy decisions in the state space. In this paper, we investigate an alternative approach that uses domain knowledge to quantify the risk in the proximity of such states by defining a violation metric.
  arXiv  Detail & Related papers  (2023-02-20T15:24:06Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Near-Optimal Multi-Agent Learning for Safe Coverage Control [76.99020416197631]
  In multi-agent coverage control problems, agents navigate their environment to reach locations that maximize the coverage of some density. In this paper, we aim to efficiently learn the density to approximately solve the coverage problem while preserving the agents' safety. We give first of its kind results: near optimal coverage in finite time while provably guaranteeing safety.
  arXiv  Detail & Related papers  (2022-10-12T16:33:34Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Constrained Markov Decision Processes via Backward Value Functions [43.649330976089004]
  We model the problem of learning with constraints as a Constrained Markov Decision Process. A key contribution of our approach is to translate cumulative cost constraints into state-based constraints. We provide theoretical guarantees under which the agent converges while ensuring safety over the course of training.
  arXiv  Detail & Related papers  (2020-08-26T20:56:16Z)
• Provably Safe PAC-MDP Exploration Using Analogies [87.41775218021044]
  Key challenge in applying reinforcement learning to safety-critical domains is understanding how to balance exploration and safety. We propose Analogous Safe-state Exploration (ASE), an algorithm for provably safe exploration in MDPs with unknown, dynamics. Our method exploits analogies between state-action pairs to safely learn a near-optimal policy in a PAC-MDP sense.
  arXiv  Detail & Related papers  (2020-07-07T15:50:50Z)
• Verifiably Safe Exploration for End-to-End Reinforcement Learning [17.401496872603943]
  This paper contributes a first approach toward enforcing formal safety constraints on end-to-end policies with visual inputs. It is evaluated on a novel benchmark that emphasizes the challenge of safely exploring in the presence of hard constraints.
  arXiv  Detail & Related papers  (2020-07-02T16:12:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.