Reliability Assessment and Safety Arguments for Machine Learning Components in Assuring Learning-Enabled Autonomous Systems

• URL: http://arxiv.org/abs/2112.00646v1
• Date: Tue, 30 Nov 2021 14:39:22 GMT
• Title: Reliability Assessment and Safety Arguments for Machine Learning Components in Assuring Learning-Enabled Autonomous Systems
• Authors: Xingyu Zhao, Wei Huang, Vibhav Bharti, Yi Dong, Victoria Cox, Alec Banks, Sen Wang, Sven Schewe, Xiaowei Huang
• Abstract summary: We present an overall assurance framework for Learning-Enabled Systems (LES) We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM.
• Score: 19.65793237440738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increasing use of Machine Learning (ML) components embedded in autonomous systems -- so-called Learning-Enabled Systems (LES) -- has resulted in the pressing need to assure their functional safety. As for traditional functional safety, the emerging consensus within both, industry and academia, is to use assurance cases for this purpose. Typically assurance cases support claims of reliability in support of safety, and can be viewed as a structured way of organising arguments and evidence generated from safety analysis and reliability modelling activities. While such assurance activities are traditionally guided by consensus-based standards developed from vast engineering experience, LES pose new challenges in safety-critical application due to the characteristics and design of ML models. In this article, we first present an overall assurance framework for LES with an emphasis on quantitative aspects, e.g., breaking down system-level safety targets to component-level requirements and supporting claims stated in reliability metrics. We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers that utilises the operational profile and robustness verification evidence. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM and propose practical solutions. Probabilistic safety arguments at the lower ML component-level are also developed based on the RAM. Finally, to evaluate and demonstrate our methods, we not only conduct experiments on synthetic/benchmark datasets but also demonstrate the scope of our methods with a comprehensive case study on Autonomous Underwater Vehicles in simulation.

Related papers

• Internal Activation as the Polar Star for Steering Unsafe LLM Behavior [50.463399903987245]
  We introduce SafeSwitch, a framework that dynamically regulates unsafe outputs by monitoring and utilizing the model's internal states. Our empirical results show that SafeSwitch reduces harmful outputs by over 80% on safety benchmarks while maintaining strong utility.
  arXiv  Detail & Related papers  (2025-02-03T04:23:33Z)
• Safety Monitoring of Machine Learning Perception Functions: a Survey [7.193217430660011]
  New dependability challenges arise when Machine Learning predictions are used in safety-critical applications. The use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system. This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context.
  arXiv  Detail & Related papers  (2024-12-09T10:58:50Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS) [16.579772998870233]
  We introduce a methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) AMLAS comprises a set of safety case patterns and a process for integrating safety assurance into the development of ML components.
  arXiv  Detail & Related papers  (2021-02-02T15:41:57Z)
• Towards a Robust and Trustworthy Machine Learning System Development [0.09236074230806578]
  We present our recent survey on the state-of-the-art ML trustworthiness and technologies from a security engineering perspective. We then push our studies forward above and beyond a survey by describing a metamodel we created that represents the body of knowledge in a standard and visualized way for ML practitioners. We propose future research directions motivated by our findings to advance the development of robust and trustworthy ML systems.
  arXiv  Detail & Related papers  (2021-01-08T14:43:58Z)
• Evaluating the Safety of Deep Reinforcement Learning Models using Semi-Formal Verification [81.32981236437395]
  We present a semi-formal verification approach for decision-making tasks based on interval analysis. Our method obtains comparable results over standard benchmarks with respect to formal verifiers. Our approach allows to efficiently evaluate safety properties for decision-making models in practical applications.
  arXiv  Detail & Related papers  (2020-10-19T11:18:06Z)
• Safety design concepts for statistical machine learning components toward accordance with functional safety standards [0.38073142980732994]
  In recent years, curial incidents and accidents have been reported due to misjudgment of statistical machine learning. In this paper, we organize five kinds of technical safety concepts (TSCs) for components toward accordance with functional safety standards.
  arXiv  Detail & Related papers  (2020-08-04T01:01:00Z)
• Quantifying Assurance in Learning-enabled Systems [3.0938904602244355]
  Dependability assurance of systems embedding machine learning components is a key step for their use in safety-critical applications. This paper develops a quantitative notion of assurance that an LES is dependable, as a core component of its assurance case. We illustrate the utility of assurance measures by application to a real world autonomous aviation system.
  arXiv  Detail & Related papers  (2020-06-18T08:11:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.