Adaptive GR(1) Specification Repair for Liveness-Preserving Shielding in Reinforcement Learning

• URL: http://arxiv.org/abs/2511.02605v1
• Date: Tue, 04 Nov 2025 14:27:28 GMT
• Title: Adaptive GR(1) Specification Repair for Liveness-Preserving Shielding in Reinforcement Learning
• Authors: Tiberiu-Andrei Georgescu, Alexander W. Goodall, Dalal Alrajeh, Francesco Belardinelli, Sebastian Uchitel,
• Abstract summary: Shielding is widely used to enforce safety in reinforcement learning (RL)<n>We develop the first adaptive shielding framework based on Generalized Reactivity of rank 1 (GR(1)) specifications.<n>Our method detects environment assumption violations at runtime and employs Inductive Logic Programming (ILP) to automatically repair GR(1) specifications online.
• Score: 46.90899478779653
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Shielding is widely used to enforce safety in reinforcement learning (RL), ensuring that an agent's actions remain compliant with formal specifications. Classical shielding approaches, however, are often static, in the sense that they assume fixed logical specifications and hand-crafted abstractions. While these static shields provide safety under nominal assumptions, they fail to adapt when environment assumptions are violated. In this paper, we develop the first adaptive shielding framework - to the best of our knowledge - based on Generalized Reactivity of rank 1 (GR(1)) specifications, a tractable and expressive fragment of Linear Temporal Logic (LTL) that captures both safety and liveness properties. Our method detects environment assumption violations at runtime and employs Inductive Logic Programming (ILP) to automatically repair GR(1) specifications online, in a systematic and interpretable way. This ensures that the shield evolves gracefully, ensuring liveness is achievable and weakening goals only when necessary. We consider two case studies: Minepump and Atari Seaquest; showing that (i) static symbolic controllers are often severely suboptimal when optimizing for auxiliary rewards, and (ii) RL agents equipped with our adaptive shield maintain near-optimal reward and perfect logical compliance compared with static shields.

Related papers

• Improving LLM Reliability through Hybrid Abstention and Adaptive Detection [1.9495934446083012]
  Large Language Models (LLMs) deployed in production environments face a fundamental safety-utility trade-off.<n>Conventional guardrails based on static rules or fixed confidence thresholds are typically context-insensitive and computationally expensive.<n>We introduce an adaptive abstention system that dynamically adjusts safety thresholds based on real-time contextual signals.
  arXiv  Detail & Related papers  (2026-02-17T07:00:09Z)
• Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
  Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
  arXiv  Detail & Related papers  (2026-02-12T22:03:35Z)
• Alignment-Aware Model Adaptation via Feedback-Guided Optimization [27.93864970404945]
  Fine-tuning is the primary mechanism for adapting foundation models to downstream tasks.<n>We propose an alignment-aware fine-tuning framework that integrates feedback from an external alignment signal through policy-gradient-based regularization.
  arXiv  Detail & Related papers  (2026-02-02T16:03:16Z)
• Q-realign: Piggybacking Realignment on Quantization for Safe and Efficient LLM Deployment [55.14890249389052]
  Existing defenses either embed safety recovery into fine-tuning or rely on fine-tuning-derived priors for post-hoc correction.<n>We propose textttQ-realign, a post-hoc defense method based on post-training quantization.<n>Our work provides a practical, turnkey solution for safety-aware deployment.
  arXiv  Detail & Related papers  (2026-01-13T00:07:24Z)
• UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models [67.91151588917396]
  Large Language Models (LLMs) have achieved remarkable progress across a wide range of tasks, but remain vulnerable to safety risks such as harmful content generation and jailbreak attacks.<n>We propose UpSafe$circ$C, a unified framework for enhancing LLM safety through safety-aware upcycling.<n>Our results highlight a new direction for LLM safety: moving from static alignment toward dynamic, modular, and inference-aware control.
  arXiv  Detail & Related papers  (2025-10-02T16:43:33Z)
• LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generation [4.29885665563186]
  LATENTGUARD is a framework that combines behavioral alignment with supervised latent space control for interpretable and precise safety steering.<n>Our results show significant improvements in both safety controllability and response interpretability without compromising utility.
  arXiv  Detail & Related papers  (2025-09-24T07:31:54Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• Probing the Robustness of Large Language Models Safety to Latent Perturbations [30.16804362984161]
  Safety alignment is a key requirement for building reliable Artificial General Intelligence.<n>We observe that minor latent shifts can still trigger unsafe responses in aligned models.<n>We introduce Layer-wise Adversarial Patch Training(LAPT), a fine-tuning strategy that injects controlled perturbations into hidden representations during training.
  arXiv  Detail & Related papers  (2025-06-19T07:03:05Z)
• Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank [64.44255178199846]
  We generalize the existing safe CLTR approach to make it applicable to state-of-the-art doubly robust CLTR. We also propose a novel approach, proximal ranking policy optimization (PRPO), that provides safety in deployment without assumptions about user behavior. PRPO is the first method with unconditional safety in deployment that translates to robust safety for real-world applications.
  arXiv  Detail & Related papers  (2024-07-29T12:23:59Z)
• Shielded Reinforcement Learning for Hybrid Systems [1.0485739694839669]
  Reinforcement learning has been leveraged to construct near-optimal controllers, but their behavior is not guaranteed to be safe. One way of imposing safety to a learned controller is to use a shield, which is correct by design. We propose the construction of a shield using the so-called barbaric method, where an approximate finite representation of an underlying partition-based two-player safety game is extracted.
  arXiv  Detail & Related papers  (2023-08-28T09:04:52Z)
• Approximate Model-Based Shielding for Safe Reinforcement Learning [83.55437924143615]
  We propose a principled look-ahead shielding algorithm for verifying the performance of learned RL policies. Our algorithm differs from other shielding approaches in that it does not require prior knowledge of the safety-relevant dynamics of the system. We demonstrate superior performance to other safety-aware approaches on a set of Atari games with state-dependent safety-labels.
  arXiv  Detail & Related papers  (2023-07-27T15:19:45Z)
• Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learning [7.103977648997475]
  Multi-Agent Reinforcement Learning (MARL) discovers policies that maximize reward but do not have safety guarantees during the learning and deployment phases. Model-based Dynamic Shielding (MBDS) to support MARL algorithm design.
  arXiv  Detail & Related papers  (2023-04-13T06:08:10Z)
• Safe Reinforcement Learning via Shielding for POMDPs [29.058332307331785]
  Reinforcement learning (RL) in safety-critical environments requires an agent to avoid decisions with catastrophic consequences. We propose and thoroughly evaluate a tight integration of formally-verified shields for POMDPs with state-of-the-art deep RL algorithms. We empirically demonstrate that an RL agent using a shield, beyond being safe, converges to higher values of expected reward.
  arXiv  Detail & Related papers  (2022-04-02T03:51:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.