COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection

• URL: http://arxiv.org/abs/2606.04749v1
• Date: Wed, 03 Jun 2026 11:30:10 GMT
• Title: COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection
• Abstract summary: In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles.<n>This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates.<n>We propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation.
• Score: 13.702950888176213
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safe robot control requires maximizing return while satisfying safety constraints. In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles, with uncertainty handled independently for each objective. This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates, thereby reducing sample efficiency. To address this issue, we propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation. COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization. COP-Q incurs minimal computational overhead and is readily compatible with most existing deep Q-learning frameworks. Experiments on robot locomotion in Brax and safe navigation in Safety-Gymnasium, covering both hard- and soft-safety settings, demonstrate that COP-Q achieves strong safety performance together with competitive or improved sample efficiency relative to representative baselines.

Related papers

• Selective Safety Steering via Value-Filtered Decoding [54.87935112120107]
  Large language models (LLMs) are trained to align with human values, but their generations may still violate safety constraints.<n>Existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model.<n>We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses.
  arXiv  Detail & Related papers  (2026-05-14T12:13:08Z)
• SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
  Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
  arXiv  Detail & Related papers  (2026-04-04T04:29:11Z)
• BarrierSteer: LLM Safety via Learning Barrier Steering [83.12893815611052]
  BarrierSteer is a novel framework that formalizes safety by embedding learned non-linear safety constraints directly into the model's latent representation space.<n>We show that BarrierSteer substantially reduces adversarial success rates, decreases unsafe generations, and outperforms existing methods.
  arXiv  Detail & Related papers  (2026-02-23T18:19:46Z)
• How to Train Your Latent Control Barrier Function: Smooth Safety Filtering Under Hard-to-Model Constraints [21.03977709777739]
  We show that reachability value functions can be adapted to be control barrier functions (CBFs) for smooth optimization-based filtering.<n>We propose the LatentCBF, which addresses both challenges through gradient penalties that lead to smooth margin functions without additional labeling.<n>Experiments on simulated benchmarks and hardware with a vision-based manipulation policy demonstrate that LatentCBF enables smooth safety filtering.
  arXiv  Detail & Related papers  (2025-11-23T20:15:28Z)
• ORN-CBF: Learning Observation-conditioned Residual Neural Control Barrier Functions via Hypernetworks [2.7458654644827134]
  Control barrier functions (CBFs) have been demonstrated as an effective method for safety-critical control of autonomous systems.<n>We propose observation-conditioned neural CBFs based on Hamilton-Jacobi (HJ) reachability analysis.
  arXiv  Detail & Related papers  (2025-09-20T10:29:36Z)
• CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention [68.95008546581339]
  Existing decoding-time interventions, such as Contrastive Decoding, often force a severe trade-off between safety and response quality.<n>We propose CARE, a novel framework for decoding-time safety alignment that integrates three key components.<n>The framework achieves a superior balance of safety, quality, and efficiency, attaining a low harmful response rate and minimal disruption to the user experience.
  arXiv  Detail & Related papers  (2025-09-01T04:50:02Z)
• Verifiable Safety Q-Filters via Hamilton-Jacobi Reachability and Multiplicative Q-Networks [8.042618833885168]
  We introduce a verifiable model-free safety filter based on Hamilton-Jacobi reachability analysis.<n>Our proposed approach successfully synthesizes formally verified, model-free safety certificates across four standard safe-control benchmarks.
  arXiv  Detail & Related papers  (2025-05-27T18:12:50Z)
• A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems [8.347548017994178]
  Safety and performance could be competing objectives, which makes their co-optimization difficult.<n>We propose a state-constrained optimal control problem, where performance objectives are encoded via a cost function and safety requirements are imposed as state constraints.<n>We demonstrate that the resultant value function satisfies a Hamilton-Jacobi-Bellman equation, which we approximate efficiently using a novel machine learning framework.
  arXiv  Detail & Related papers  (2025-02-16T09:46:17Z)
• A Multiplicative Value Function for Safe and Efficient Reinforcement Learning [131.96501469927733]
  We propose a safe model-free RL algorithm with a novel multiplicative value function consisting of a safety critic and a reward critic. The safety critic predicts the probability of constraint violation and discounts the reward critic that only estimates constraint-free returns. We evaluate our method in four safety-focused environments, including classical RL benchmarks augmented with safety constraints and robot navigation tasks with images and raw Lidar scans as observations.
  arXiv  Detail & Related papers  (2023-03-07T18:29:15Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Safe Wasserstein Constrained Deep Q-Learning [2.088376060651494]
  This paper presents a distributionally robust Q-Learning algorithm (DrQ) which leverages Wasserstein ambiguity sets to provide idealistic probabilistic out-of-sample safety guarantees. Using a case study of lithium-ion battery fast charging, we explore how idealistic safety guarantees translate to generally improved safety.
  arXiv  Detail & Related papers  (2020-02-07T21:23:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.