Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learning

• URL: http://arxiv.org/abs/2502.16328v2
• Date: Sat, 14 Jun 2025 12:48:51 GMT
• Title: Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learning
• Authors: Zahra Shahrooei, Ali Baheri,
• Abstract summary: This paper views safety as taking actions with more predictable consequences under environmentity.<n>By integrating this uncertainty score into the decision-making objective, the agent is encouraged to favor actions with more predictable outcomes.
• Score: 4.14360329494344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The primary goal of reinforcement learning is to develop decision-making policies that prioritize optimal performance, frequently without considering safety. In contrast, safe reinforcement learning seeks to reduce or avoid unsafe behavior. This paper views safety as taking actions with more predictable consequences under environment stochasticity and introduces a temporal difference algorithm that uses optimal transport theory to quantify the uncertainty associated with actions. By integrating this uncertainty score into the decision-making objective, the agent is encouraged to favor actions with more predictable outcomes. We theoretically prove that our algorithm leads to a reduction in the probability of visiting unsafe states. We evaluate the proposed algorithm on several case studies in the presence of various forms of environment uncertainty. The results demonstrate that our method not only provides safer behavior but also maintains the performance. A Python implementation of our algorithm is available at \href{https://github.com/SAILRIT/Risk-averse-TD-Learning}{https://github.com/SAILRIT/OT-guided-TD-Learning}.

Related papers

• Risk-averse learning with delayed feedback [17.626195546400247]
  We develop two risk-averse learning algorithms that rely on one-point and two-point zeroth-order optimization approaches. The results suggest that the two-point risk-averse learning achieves a smaller regret bound than the one-point algorithm.
  arXiv  Detail & Related papers  (2024-09-25T12:32:22Z)
• Optimal Transport-Assisted Risk-Sensitive Q-Learning [4.14360329494344]
  This paper presents a risk-sensitive Q-learning algorithm that leverages optimal transport theory to enhance the agent safety. We validate the proposed algorithm in a Gridworld environment.
  arXiv  Detail & Related papers  (2024-06-17T17:32:25Z)
• Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time [0.6554326244334868]
  We present an online reinforcement learning algorithm for constrained Markov decision processes with a safety constraint. We show that the learned policy is safe with high confidence. We also demonstrate that efficient exploration can be achieved by defining a subset of the state-space called proxy set.
  arXiv  Detail & Related papers  (2024-03-23T20:22:30Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization [1.3597551064547502]
  This study introduces a novel safe reinforcement learning algorithm, Safety Critic Policy Optimization. In this study, we define the safety critic, a mechanism that nullifies rewards obtained through violating safety constraints. Our theoretical analysis indicates that the proposed algorithm can automatically balance the trade-off between adhering to safety constraints and maximizing rewards.
  arXiv  Detail & Related papers  (2023-11-01T22:12:50Z)
• Approximate Shielding of Atari Agents for Safe Exploration [83.55437924143615]
  We propose a principled algorithm for safe exploration based on the concept of shielding. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations.
  arXiv  Detail & Related papers  (2023-04-21T16:19:54Z)
• Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning [72.97229770329214]
  We introduce a general approach for seeking high dimensional non-linear optimization problems in which maintaining safety during learning is crucial. Our approach called LBSGD is based on applying a logarithmic barrier approximation with a carefully chosen step size. We demonstrate the effectiveness of our approach on minimizing violation in policy tasks in safe reinforcement learning.
  arXiv  Detail & Related papers  (2022-07-21T11:14:47Z)
• Safe Reinforcement Learning by Imagining the Near Future [37.0376099401243]
  In this work, we focus on the setting where unsafe states can be avoided by planning ahead a short time into the future. We devise a model-based algorithm that heavily penalizes unsafe trajectories, and derive guarantees that our algorithm can avoid unsafe states under certain assumptions. Experiments demonstrate that our algorithm can achieve competitive rewards with fewer safety violations in several continuous control tasks.
  arXiv  Detail & Related papers  (2022-02-15T23:28:24Z)
• Surveillance Evasion Through Bayesian Reinforcement Learning [78.79938727251594]
  We consider a 2D continuous path planning problem with a completely unknown intensity of random termination. Those Observers' surveillance intensity is a priori unknown and has to be learned through repetitive path planning.
  arXiv  Detail & Related papers  (2021-09-30T02:29:21Z)
• Improving Safety in Deep Reinforcement Learning using Unsupervised Action Planning [4.2955354157580325]
  One of the key challenges to deep reinforcement learning (deep RL) is to ensure safety at both training and testing phases. We propose a novel technique of unsupervised action planning to improve the safety of on-policy reinforcement learning algorithms. Our results show that the proposed safety RL algorithm can achieve higher rewards compared with multiple baselines in both discrete and continuous control problems.
  arXiv  Detail & Related papers  (2021-09-29T10:26:29Z)
• Policy Gradient Bayesian Robust Optimization for Imitation Learning [49.881386773269746]
  We derive a novel policy gradient-style robust optimization approach, PG-BROIL, to balance expected performance and risk. Results suggest PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse.
  arXiv  Detail & Related papers  (2021-06-11T16:49:15Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments [24.75527261989899]
  Safety is a critical concern when deploying reinforcement learning agents for realistic tasks. We propose the context-aware safe reinforcement learning (CASRL) method to realize safe adaptation in non-stationary environments. Results show that the proposed algorithm significantly outperforms existing baselines in terms of safety and robustness.
  arXiv  Detail & Related papers  (2021-01-02T23:52:22Z)
• Bayesian Robust Optimization for Imitation Learning [34.40385583372232]
  Inverse reinforcement learning can enable generalization to new states by learning a parameterized reward function. Existing safe imitation learning approaches based on IRL deal with this uncertainty using a maxmin framework. BROIL provides a natural way to interpolate between return-maximizing and risk-minimizing behaviors.
  arXiv  Detail & Related papers  (2020-07-24T01:52:11Z)
• Learning Bounds for Risk-sensitive Learning [86.50262971918276]
  In risk-sensitive learning, one aims to find a hypothesis that minimizes a risk-averse (or risk-seeking) measure of loss. We study the generalization properties of risk-sensitive learning schemes whose optimand is described via optimized certainty equivalents.
  arXiv  Detail & Related papers  (2020-06-15T05:25:02Z)
• Excursion Search for Constrained Bayesian Optimization under a Limited Budget of Failures [62.41541049302712]
  We propose a novel decision maker grounded in control theory that controls the amount of risk we allow in the search as a function of a given budget of failures. Our algorithm uses the failures budget more efficiently in a variety of optimization experiments, and generally achieves lower regret, than state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-05-15T09:54:09Z)
• Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems [81.7983463275447]
  Learning-based control algorithms require data collection with abundant supervision for training. We present a new approach for optimal motion planning with safe exploration that integrates chance-constrained optimal control with dynamics learning and feedback control.
  arXiv  Detail & Related papers  (2020-05-09T05:57:43Z)
• Cautious Reinforcement Learning with Logical Constraints [78.96597639789279]
  An adaptive safe padding forces Reinforcement Learning (RL) to synthesise optimal control policies while ensuring safety during the learning process. Theoretical guarantees are available on the optimality of the synthesised policies and on the convergence of the learning algorithm.
  arXiv  Detail & Related papers  (2020-02-26T00:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.