Conformal Safety Shielding for Imperfect-Perception Agents

• URL: http://arxiv.org/abs/2506.17275v2
• Date: Sat, 26 Jul 2025 17:30:59 GMT
• Title: Conformal Safety Shielding for Imperfect-Perception Agents
• Authors: William Scarbro, Calum Imrie, Sinem Getir Yaman, Kavan Fatehi, Corina S. Pasareanu, Radu Calinescu, Ravi Mangal,
• Abstract summary: We consider the problem of safe control in discrete autonomous agents that use learned components for imperfect perception.<n>We propose a shield construction that provides run-time safety guarantees under perception errors.
• Score: 7.5422935754618825
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider the problem of safe control in discrete autonomous agents that use learned components for imperfect perception (or more generally, state estimation) from high-dimensional observations. We propose a shield construction that provides run-time safety guarantees under perception errors by restricting the actions available to an agent, modeled as a Markov decision process, as a function of the state estimates. Our construction uses conformal prediction for the perception component, which guarantees that for each observation, the predicted set of estimates includes the actual state with a user-specified probability. The shield allows an action only if it is allowed for all the estimates in the predicted set, resulting in local safety. We also articulate and prove a global safety property of existing shield constructions for perfect-perception agents bounding the probability of reaching unsafe states if the agent always chooses actions prescribed by the shield. We illustrate our approach with a case-study of an experimental autonomous system that guides airplanes on taxiways using high-dimensional perception DNNs.

Related papers

• Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention [65.47632669243657]
  A dishonest institution can exploit mechanisms to discriminate or unjustly deny services under the guise of uncertainty.<n>We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage.<n>We propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence.
  arXiv  Detail & Related papers  (2025-05-29T19:47:50Z)
• Realizable Continuous-Space Shields for Safe Reinforcement Learning [13.728961635717134]
  We present the first shielding approach specifically designed to ensure the satisfaction of safety requirements in continuous state and action spaces.<n>Our method builds upon realizability, an essential property that confirms the shield will always be able to generate a safe action for any state in the environment.
  arXiv  Detail & Related papers  (2024-10-02T21:08:11Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Approximate Shielding of Atari Agents for Safe Exploration [83.55437924143615]
  We propose a principled algorithm for safe exploration based on the concept of shielding. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations.
  arXiv  Detail & Related papers  (2023-04-21T16:19:54Z)
• Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling [0.0]
  We demonstrate the use of the conformal prediction framework to construct reliable predictors for detecting railway signals. Our approach is based on a novel dataset that includes images taken from the perspective of a train operator and state-of-the-art object detectors.
  arXiv  Detail & Related papers  (2023-04-12T08:10:13Z)
• Safe Perception-Based Control under Stochastic Sensor Uncertainty using Conformal Prediction [27.515056747751053]
  We propose a perception-based control framework that quantifies estimation uncertainty of perception maps. We also integrate these uncertainty representations into the control design. We demonstrate the effectiveness of our proposed perception-based controller for a LiDAR-enabled F1/10th car.
  arXiv  Detail & Related papers  (2023-04-01T01:45:53Z)
• USC: Uncompromising Spatial Constraints for Safety-Oriented 3D Object Detectors in Autonomous Driving [7.355977594790584]
  We consider the safety-oriented performance of 3D object detectors in autonomous driving contexts.<n>We present uncompromising spatial constraints (USC), which characterize a simple yet important localization requirement.<n>We incorporate the quantitative measures into common loss functions to enable safety-oriented fine-tuning for existing models.
  arXiv  Detail & Related papers  (2022-09-21T14:03:08Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)
• Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty [54.88405167739227]
  We present HAICU, a method for heterogeneous-agent trajectory forecasting that explicitly incorporates agents' class probabilities. We additionally present PUP, a new challenging real-world autonomous driving dataset. We demonstrate that incorporating class probabilities in trajectory forecasting significantly improves performance in the face of uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T10:28:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.