Global Safe Sequential Learning via Efficient Knowledge Transfer

• URL: http://arxiv.org/abs/2402.14402v2
• Date: Mon, 15 Apr 2024 16:57:36 GMT
• Title: Global Safe Sequential Learning via Efficient Knowledge Transfer
• Authors: Cen-You Li, Olaf Duennbier, Marc Toussaint, Barbara Rakitsch, Christoph Zimmer,
• Abstract summary: In many medical or engineering applications, the data selection is constrained by a priori unknown safety conditions. We propose transfer safe sequential learning to accelerate the learning of safety.
• Score: 21.817220232038157
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequential learning methods such as active learning and Bayesian optimization select the most informative data to learn about a task. In many medical or engineering applications, the data selection is constrained by a priori unknown safety conditions. A promissing line of safe learning methods utilize Gaussian processes (GPs) to model the safety probability and perform data selection in areas with high safety confidence. However, accurate safety modeling requires prior knowledge or consumes data. In addition, the safety confidence centers around the given observations which leads to local exploration. As transferable source knowledge is often available in safety critical experiments, we propose to consider transfer safe sequential learning to accelerate the learning of safety. We further consider a pre-computation of source components to reduce the additional computational load that is introduced by incorporating source data. In this paper, we theoretically analyze the maximum explorable safe regions of conventional safe learning methods. Furthermore, we empirically demonstrate that our approach 1) learns a task with lower data consumption, 2) globally explores multiple disjoint safe regions under guidance of the source knowledge, and 3) operates with computation comparable to conventional safe learning methods.

Related papers

• Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints [15.904640266226023]
  We design a safety model that performs credit assignment to assess contributions of partial state-action trajectories on safety. We derive an effective algorithm for optimizing a safe policy using the learned safety model. We devise a method to dynamically adapt the tradeoff coefficient between safety reward and safety compliance.
  arXiv  Detail & Related papers  (2024-05-05T17:27:22Z)
• Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version) [1.279257604152629]
  Safe exploration aims at addressing the limitations of Reinforcement Learning (RL) in safety-critical scenarios. Several methods exist to incorporate external knowledge or to use sensor data to limit the exploration of unsafe states. In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement.
  arXiv  Detail & Related papers  (2023-07-10T22:28:33Z)
• Approximate Shielding of Atari Agents for Safe Exploration [83.55437924143615]
  We propose a principled algorithm for safe exploration based on the concept of shielding. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations.
  arXiv  Detail & Related papers  (2023-04-21T16:19:54Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning [72.97229770329214]
  We introduce a general approach for seeking high dimensional non-linear optimization problems in which maintaining safety during learning is crucial. Our approach called LBSGD is based on applying a logarithmic barrier approximation with a carefully chosen step size. We demonstrate the effectiveness of our approach on minimizing violation in policy tasks in safe reinforcement learning.
  arXiv  Detail & Related papers  (2022-07-21T11:14:47Z)
• Active Learning with Safety Constraints [25.258564629480063]
  We investigate the complexity of learning the best safe decision in interactive environments. We propose an adaptive experimental design-based algorithm, which we show efficiently trades off between the difficulty of showing an arm is unsafe vs suboptimal.
  arXiv  Detail & Related papers  (2022-06-22T15:45:38Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)
• Safe Policy Optimization with Local Generalized Linear Function Approximations [17.84511819022308]
  Existing safe exploration methods guaranteed safety under the assumption of regularity. We propose a novel algorithm, SPO-LF, that optimize an agent's policy while learning the relation between a locally available feature obtained by sensors and environmental reward/safety. We experimentally show that our algorithm is 1) more efficient in terms of sample complexity and computational cost and 2) more applicable to large-scale problems than previous safe RL methods with theoretical guarantees.
  arXiv  Detail & Related papers  (2021-11-09T00:47:50Z)
• Data Generation Method for Learning a Low-dimensional Safe Region in Safe Reinforcement Learning [9.903083270841638]
  Safe reinforcement learning aims to learn a control policy while ensuring that neither the system nor the environment gets damaged during the learning process. For implementing safe reinforcement learning on highly nonlinear and high-dimensional dynamical systems, one possible approach is to find a low-dimensional safe region via data-driven feature extraction methods. As the reliability of the learned safety estimates is data-dependent, we investigate in this work how different training data will affect the safe reinforcement learning approach.
  arXiv  Detail & Related papers  (2021-09-10T19:22:43Z)
• Learning to be Safe: Deep RL with a Safety Critic [72.00568333130391]
  A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. We propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors.
  arXiv  Detail & Related papers  (2020-10-27T20:53:20Z)
• Provably Safe PAC-MDP Exploration Using Analogies [87.41775218021044]
  Key challenge in applying reinforcement learning to safety-critical domains is understanding how to balance exploration and safety. We propose Analogous Safe-state Exploration (ASE), an algorithm for provably safe exploration in MDPs with unknown, dynamics. Our method exploits analogies between state-action pairs to safely learn a near-optimal policy in a PAC-MDP sense.
  arXiv  Detail & Related papers  (2020-07-07T15:50:50Z)
• Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems [81.7983463275447]
  Learning-based control algorithms require data collection with abundant supervision for training. We present a new approach for optimal motion planning with safe exploration that integrates chance-constrained optimal control with dynamics learning and feedback control.
  arXiv  Detail & Related papers  (2020-05-09T05:57:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.