Bootstrapping confidence in future safety based on past safe operation

• URL: http://arxiv.org/abs/2110.10718v1
• Date: Wed, 20 Oct 2021 18:36:23 GMT
• Title: Bootstrapping confidence in future safety based on past safe operation
• Authors: Peter Bishop, Andrey Povyakalo and Lorenzo Strigini
• Abstract summary: We show an approach to confidence of low enough probability of causing accidents in the early phases of operation. This formalises the common approach of operating a system on a limited basis in the hope that mishap-free operation will confirm one's confidence in its safety.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: With autonomous vehicles (AVs), a major concern is the inability to give meaningful quantitative assurance of safety, to the extent required by society - e.g. that an AV must be at least as safe as a good human driver - before that AV is in extensive use. We demonstrate an approach to achieving more moderate, but useful, confidence, e.g., confidence of low enough probability of causing accidents in the early phases of operation. This formalises mathematically the common approach of operating a system on a limited basis in the hope that mishap-free operation will confirm one's confidence in its safety and allow progressively more extensive operation: a process of "bootstrapping" of confidence. Translating that intuitive approach into theorems shows: (1) that it is substantially sound in the right circumstances, and could be a good method for deciding about the early deployment phase for an AV; (2) how much confidence can be rightly derived from such a "cautious deployment" approach, so that we can avoid over-optimism; (3) under which conditions our sound formulas for future confidence are applicable; (4) thus, which analyses of the concrete situations, and/or constraints on practice, are needed in order to enjoy the advantages of provably correct confidence in adequate future safety.

Related papers

• On the Robustness of Adversarial Training Against Uncertainty Attacks [9.180552487186485]
  In learning problems, the noise inherent to the task at hand hinders the possibility to infer without a certain degree of uncertainty. In this work, we reveal both empirically and theoretically that defending against adversarial examples, i.e., carefully perturbed samples that cause misclassification, guarantees a more secure, trustworthy uncertainty estimate. To support our claims, we evaluate multiple adversarial-robust models from the publicly available benchmark RobustBench on the CIFAR-10 and ImageNet datasets.
  arXiv  Detail & Related papers  (2024-10-29T11:12:44Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversariality [13.240598841087841]
  We introduce FREA, a novel safety-critical scenarios generation method that incorporates the Largest Feasible Region (LFR) of AV as guidance. Experiments illustrate that FREA can effectively generate safety-critical scenarios, yielding considerable near-miss events.
  arXiv  Detail & Related papers  (2024-06-05T06:26:15Z)
• A Counterfactual Safety Margin Perspective on the Scoring of Autonomous Vehicles' Riskiness [52.27309191283943]
  This paper presents a data-driven framework for assessing the risk of different AVs' behaviors. We propose the notion of counterfactual safety margin, which represents the minimum deviation from nominal behavior that could cause a collision.
  arXiv  Detail & Related papers  (2023-08-02T09:48:08Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Building a Credible Case for Safety: Waymo's Approach for the Determination of Absence of Unreasonable Risk [2.2386635730984117]
  A safety case for fully autonomous operations is a formal way to explain how a company determines that an AV system is safe. It involves an explanation of the system, the methodologies used to develop it, the metrics used to validate it and the actual results of validation tests. This paper helps enabling such alignment by providing foundational thinking into how a system is determined to be ready for deployment.
  arXiv  Detail & Related papers  (2023-06-02T21:05:39Z)
• Did You Mean...? Confidence-based Trade-offs in Semantic Parsing [52.28988386710333]
  We show how a calibrated model can help balance common trade-offs in task-oriented parsing. We then examine how confidence scores can help optimize the trade-off between usability and safety.
  arXiv  Detail & Related papers  (2023-03-29T17:07:26Z)
• Conservative Safety Critics for Exploration [120.73241848565449]
  We study the problem of safe exploration in reinforcement learning (RL) We learn a conservative safety estimate of environment states through a critic. We show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates.
  arXiv  Detail & Related papers  (2020-10-27T17:54:25Z)
• Towards Safe Policy Improvement for Non-Stationary MDPs [48.9966576179679]
  Many real-world problems of interest exhibit non-stationarity, and when stakes are high, the cost associated with a false stationarity assumption may be unacceptable. We take the first steps towards ensuring safety, with high confidence, for smoothly-varying non-stationary decision problems. Our proposed method extends a type of safe algorithm, called a Seldonian algorithm, through a synthesis of model-free reinforcement learning with time-series analysis.
  arXiv  Detail & Related papers  (2020-10-23T20:13:51Z)
• Positive Trust Balance for Self-Driving Car Deployment [3.106768467227812]
  Decision about when self-driving cars are ready to deploy is likely to be made with insufficient lagging metric data. A Positive Trust Balance approach can help with making a responsible deployment decision.
  arXiv  Detail & Related papers  (2020-09-12T14:23:47Z)
• Assessing Safety-Critical Systems from Operational Testing: A Study on Autonomous Vehicles [3.629865579485447]
  Demonstrating high reliability and safety for safety-critical systems (SCSs) remains a hard problem. We use Autonomous Vehicles (AVs) as a current example to revisit the problem of demonstrating high reliability.
  arXiv  Detail & Related papers  (2020-08-19T19:50:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.