Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

• URL: http://arxiv.org/abs/2608.26571v1
• Date: Thu, 27 Aug 2026 03:24:56 GMT
• Title: Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals
• Abstract summary: Contrastive reinforcement learning scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective.<n>Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values.<n>We introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization.
• Score: 1.7496207597922033
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective. However, in a failure-terminated Markov decision process, established CRL considers pre-failure future goals only when constructing positive samples, without accounting for the probability mass removed by failure termination. Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values. Consequently, near-failure trajectories provide disproportionately strong supervision of success despite retaining little future occupancy. Unsafe actions can thereby be reinforced through catastrophic failure bootstrapping, leading to failed policy learning and unsustainable goal-reaching behaviours. To address this problem, we introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization. The resulting method, Safe Contrastive Reinforcement Learning (Safe-CRL), requires only the one-bit signal provided by failure termination to scale safe goal-conditioned policy learning. Across twelve failure-prone robot navigation and locomotion tasks, Safe-CRL consistently improves survival and substantially outperforms the Scaling-CRL baseline in goal-reaching performance. Additionally, deep Safe-CRL policies exhibit complex failure-avoidance behaviours. This study completes the CRL theory under failure termination and provides a scalable safe RL framework. The code is available via https://github.com/RomainLITUD/safe-crl.

Related papers

• RL Forgets! Towards Continual Policy Optimization [57.4336338996653]
  Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks.<n>Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting.<n>We propose Continual Policy Optimization (CPO), a replay-free framework grounded in a prior-task behavioral KL objective.
  arXiv  Detail & Related papers  (2026-07-05T15:46:11Z)
• Survival Reinforcement Learning: Toward Scalable Self-Supervised RL [10.768601219140153]
  Self-supervised Contrastive Reinforcement Learning (CRL) has shown remarkable depth-scaling capabilities.<n>We introduce Survival Reinforcement Learning (SRL), an online classification-based alternative that extends the survival value learning framework.
  arXiv  Detail & Related papers  (2026-05-29T13:05:08Z)
• On Minimizing Adversarial Counterfactual Error in Adversarial RL [18.044879441434432]
  adversarial noise poses significant risks in safety-critical scenarios.<n>We introduce a novel objective called Adversarial Counterfactual Error (ACoE)<n>Our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges.
  arXiv  Detail & Related papers  (2024-06-07T08:14:24Z)
• Strategically Conservative Q-Learning [89.17906766703763]
  offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions. We propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate.
  arXiv  Detail & Related papers  (2024-06-06T22:09:46Z)
• Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy [4.854443247023496]
  offline goal-conditioned reinforcement learning (GCRL) aims at solving goal-reaching tasks with sparse rewards from an offline dataset. We propose a new method called Recovery-based Supervised Learning (RbSL) to accomplish safety-critical tasks with various goals.
  arXiv  Detail & Related papers  (2024-03-04T05:20:57Z)
• A Multiplicative Value Function for Safe and Efficient Reinforcement Learning [131.96501469927733]
  We propose a safe model-free RL algorithm with a novel multiplicative value function consisting of a safety critic and a reward critic. The safety critic predicts the probability of constraint violation and discounts the reward critic that only estimates constraint-free returns. We evaluate our method in four safety-focused environments, including classical RL benchmarks augmented with safety constraints and robot navigation tasks with images and raw Lidar scans as observations.
  arXiv  Detail & Related papers  (2023-03-07T18:29:15Z)
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning [63.53407136812255]
  Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. Existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly.
  arXiv  Detail & Related papers  (2021-05-17T20:16:46Z)
• Assured Learning-enabled Autonomy: A Metacognitive Reinforcement Learning Framework [4.427447378048202]
  Reinforcement learning (RL) agents with pre-specified reward functions cannot provide guaranteed safety across variety of circumstances. An assured autonomous control framework is presented in this paper by empowering RL algorithms with metacognitive learning capabilities.
  arXiv  Detail & Related papers  (2021-03-23T14:01:35Z)
• Conservative Safety Critics for Exploration [120.73241848565449]
  We study the problem of safe exploration in reinforcement learning (RL) We learn a conservative safety estimate of environment states through a critic. We show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates.
  arXiv  Detail & Related papers  (2020-10-27T17:54:25Z)
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations [88.94162416324505]
  A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
  arXiv  Detail & Related papers  (2020-03-19T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.