Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

• URL: http://arxiv.org/abs/2603.15136v1
• Date: Mon, 16 Mar 2026 11:28:47 GMT
• Title: Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
• Abstract summary: offline safe reinforcement learning seeks reward-maximizing policies from static datasets under strict safety constraints.<n>We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy.<n>Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.
• Score: 0.764671395172401
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. To account for finite-data approximation error in the learned safety boundary, we add a conformal prediction calibration step that adjusts the safety threshold and provides finite-sample probabilistic safety coverage. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Related papers

• Selective Safety Steering via Value-Filtered Decoding [54.87935112120107]
  Large language models (LLMs) are trained to align with human values, but their generations may still violate safety constraints.<n>Existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model.<n>We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses.
  arXiv  Detail & Related papers  (2026-05-14T12:13:08Z)
• Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning [13.674504740468159]
  We show how a safety reachability set enforces safety constraints without unstable min/max or Lagrangian optimization.<n>We also show how this set enforces safety constraints without unstable min/max or Lagrangian optimization.<n>Our method matches or outperforms state of the art baselines while maintaining safety.
  arXiv  Detail & Related papers  (2026-03-08T11:09:33Z)
• BarrierSteer: LLM Safety via Learning Barrier Steering [83.12893815611052]
  BarrierSteer is a novel framework that formalizes safety by embedding learned non-linear safety constraints directly into the model's latent representation space.<n>We show that BarrierSteer substantially reduces adversarial success rates, decreases unsafe generations, and outperforms existing methods.
  arXiv  Detail & Related papers  (2026-02-23T18:19:46Z)
• Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
  Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
  arXiv  Detail & Related papers  (2026-02-12T22:03:35Z)
• Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learning [0.0]
  We propose a framework that formulates safe offline RL as a state-constrained optimal control problem to co-optimize safety and performance.<n>EpiFlow achieves competitive returns with near-zero empirical safety violations.
  arXiv  Detail & Related papers  (2026-02-08T16:56:21Z)
• Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL [21.316556011382932]
  Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints.<n>We propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL)<n>DRCORL first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference.
  arXiv  Detail & Related papers  (2025-02-18T00:00:03Z)
• FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning [7.888219789657414]
  Safe offline reinforcement learning aims to learn policies that maximize cumulative rewards while adhering to safety constraints.<n>Key challenge is balancing safety and performance, particularly when the policy encounters out-of-distribution states and actions.<n>We introduce Feasibility Informed Advantage Weighted Actor-Critic (FAWAC), a method that prioritizes persistent safety in constrained Markov decision processes.
  arXiv  Detail & Related papers  (2024-12-12T02:28:50Z)
• Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model [23.93820548551533]
  We propose FISOR (FeasIbility-guided Safe Offline RL), which allows safety constraint adherence, reward, and offline policy learning. In FISOR, the translated optimal policy for the translated optimization problem can be derived in a special form of weighted behavior cloning. We show that FISOR is the only method that can guarantee safety satisfaction in all tasks, while achieving top returns in most tasks.
  arXiv  Detail & Related papers  (2024-01-19T14:05:09Z)
• Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments [84.3830478851369]
  We propose a safe reinforcement learning approach that can jointly learn the environment and optimize the control policy. Our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.
  arXiv  Detail & Related papers  (2022-09-29T20:49:25Z)
• Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
  We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-07-04T11:43:23Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.