Safe Continual Reinforcement Learning in Non-stationary Environments

• URL: http://arxiv.org/abs/2604.19737v1
• Date: Tue, 21 Apr 2026 17:58:32 GMT
• Title: Safe Continual Reinforcement Learning in Non-stationary Environments
• Abstract summary: Reinforcement learning (RL) offers a compelling data-driven paradigm for synthesizing controllers for complex systems when accurate physical models are unavailable.<n>Most existing control-oriented RL methods assume stationarity and, therefore, struggle in real-world non-stationary deployments.<n>We study safe continual reinforcement learning algorithms that can adapt over the system's lifetime while preserving safety.
• Score: 2.8757268399458855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) offers a compelling data-driven paradigm for synthesizing controllers for complex systems when accurate physical models are unavailable; however, most existing control-oriented RL methods assume stationarity and, therefore, struggle in real-world non-stationary deployments where system dynamics and operating conditions can change unexpectedly. Moreover, RL controllers acting in physical environments must satisfy safety constraints throughout their learning and execution phases, rendering transient violations during adaptation unacceptable. Although continual RL and safe RL have each addressed non-stationarity and safety, respectively, their intersection remains comparatively unexplored, motivating the study of safe continual RL algorithms that can adapt over the system's lifetime while preserving safety. In this work, we systematically investigate safe continual reinforcement learning by introducing three benchmark environments that capture safety-critical continual adaptation and by evaluating representative approaches from safe RL, continual RL, and their combinations. Our empirical results reveal a fundamental tension between maintaining safety constraints and preventing catastrophic forgetting under non-stationary dynamics, with existing methods generally failing to achieve both objectives simultaneously. To address this shortcoming, we examine regularization-based strategies that partially mitigate this trade-off and characterize their benefits and limitations. Finally, we outline key open challenges and research directions toward developing safe, resilient learning-based controllers capable of sustained autonomous operation in changing environments.

Related papers

• Sampling-Based Safe Reinforcement Learning [30.21184523512902]
  We propose a model-based reinforcement learning algorithm that maintains safety throughout the learning process.<n>This formulation approximates an intractable worst-case optimization over uncertain dynamics.<n>Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy.
  arXiv  Detail & Related papers  (2026-05-19T07:21:53Z)
• RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning [0.8291942198324129]
  This paper introduces Reinforcement Learning System-Theoretic Process Analysis (RL-STPA)<n> RL-STPA adapts conventional hazard analysis to address RL's unique challenges through three key contributions.<n>We demonstrate RL-STPA in the safety-critical test case of autonomous drone navigation and landing.
  arXiv  Detail & Related papers  (2026-04-16T16:27:42Z)
• Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
  Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
  arXiv  Detail & Related papers  (2026-02-12T22:03:35Z)
• Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization [47.30677525394649]
  We analyze the interplay between two well-established techniques in model-free reinforcement learning: entropy regularization and constraints penalization.<n>We show that entropy regularization in constrained RL inherently biases learning toward maximizing the number of future viable actions, thereby promoting constraints satisfaction robust to action noise.<n>We conclude that the connection between entropy regularization and robustness is a promising avenue for further empirical and theoretical investigation.
  arXiv  Detail & Related papers  (2025-06-12T16:34:19Z)
• Safely Learning Controlled Stochastic Dynamics [61.82896036131116]
  We introduce a method that ensures safe exploration and efficient estimation of system dynamics.<n>After training, the learned model enables predictions of the system's dynamics and permits safety verification of any given control.<n>We provide theoretical guarantees for safety and derive adaptive learning rates that improve with increasing Sobolev regularity of the true dynamics.
  arXiv  Detail & Related papers  (2025-06-03T11:17:07Z)
• On the Design of Safe Continual RL Methods for Control of Nonlinear Systems [2.4920094574814864]
  Reinforcement learning algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics.<n>In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in closed loops.<n>Continual adaptation on the system's safety is an understudied problem.
  arXiv  Detail & Related papers  (2025-02-21T20:34:40Z)
• Learning to explore when mistakes are not allowed [1.179778723980276]
  We propose a method that enables agents to learn goal-conditioned behaviors that explore without the risk of making harmful mistakes.<n> Exploration without risks can seem paradoxical, but environment dynamics are often uniform in space.<n>We evaluate our method in simulated environments and demonstrate that it not only provides substantial coverage of the goal space but also reduces the occurrence of mistakes to a minimum.
  arXiv  Detail & Related papers  (2025-02-19T15:11:51Z)
• ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning [48.536695794883826]
  We present ActSafe, a novel model-based RL algorithm for safe and efficient exploration.<n>We show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time.<n>In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements.
  arXiv  Detail & Related papers  (2024-10-12T10:46:02Z)
• Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems [15.863561935347692]
  We develop provably safe and convergent reinforcement learning algorithms for control of nonlinear dynamical systems. Recent advances at the intersection of control and RL follow a two-stage, safety filter approach to enforcing hard safety constraints. We develop a single-stage, sampling-based approach to hard constraint satisfaction that learns RL controllers enjoying classical convergence guarantees.
  arXiv  Detail & Related papers  (2024-03-06T19:39:20Z)
• Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments [84.3830478851369]
  We propose a safe reinforcement learning approach that can jointly learn the environment and optimize the control policy. Our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.
  arXiv  Detail & Related papers  (2022-09-29T20:49:25Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
  We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-07-04T11:43:23Z)
• Lyapunov-based uncertainty-aware safe reinforcement learning [0.0]
  InReinforcement learning (RL) has shown a promising performance in learning optimal policies for a variety of sequential decision-making tasks. In many real-world RL problems, besides optimizing the main objectives, the agent is expected to satisfy a certain level of safety. We propose a Lyapunov-based uncertainty-aware safe RL model to address these limitations.
  arXiv  Detail & Related papers  (2021-07-29T13:08:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.