Unifying Evaluation of Machine Learning Safety Monitors
- URL: http://arxiv.org/abs/2208.14660v1
- Date: Wed, 31 Aug 2022 07:17:42 GMT
- Title: Unifying Evaluation of Machine Learning Safety Monitors
- Authors: Joris Guerin and Raul Sena Ferreira and Kevin Delmas and J\'er\'emie
Guiochet
- Abstract summary: runtime monitors have been developed to detect prediction errors and keep the system in a safe state during operations.
This paper introduces three unified safety-oriented metrics, representing the safety benefits of the monitor (Safety Gain) and the remaining safety gaps after using it (Residual Hazard)
Three use-cases (classification, drone landing, and autonomous driving) are used to demonstrate how metrics from the literature can be expressed in terms of the proposed metrics.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With the increasing use of Machine Learning (ML) in critical autonomous
systems, runtime monitors have been developed to detect prediction errors and
keep the system in a safe state during operations. Monitors have been proposed
for different applications involving diverse perception tasks and ML models,
and specific evaluation procedures and metrics are used for different contexts.
This paper introduces three unified safety-oriented metrics, representing the
safety benefits of the monitor (Safety Gain), the remaining safety gaps after
using it (Residual Hazard), and its negative impact on the system's performance
(Availability Cost). To compute these metrics, one requires to define two
return functions, representing how a given ML prediction will impact expected
future rewards and hazards. Three use-cases (classification, drone landing, and
autonomous driving) are used to demonstrate how metrics from the literature can
be expressed in terms of the proposed metrics. Experimental results on these
examples show how different evaluation choices impact the perceived performance
of a monitor. As our formalism requires us to formulate explicit safety
assumptions, it allows us to ensure that the evaluation conducted matches the
high-level system requirements.
Related papers
- SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models [75.67623347512368]
We propose toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs.
Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol.
Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs.
arXiv Detail & Related papers (2024-10-24T17:14:40Z) - Learning Run-time Safety Monitors for Machine Learning Components [8.022333445774382]
This paper introduces a process for creating safety monitors for machine learning components through the use of degraded datasets and machine learning.
The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output.
arXiv Detail & Related papers (2024-06-23T21:25:06Z) - System Safety Monitoring of Learned Components Using Temporal Metric Forecasting [8.76735390039138]
In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations.
We propose a safety monitoring method based on probabilistic time series forecasting.
We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models.
arXiv Detail & Related papers (2024-05-21T23:48:26Z) - Designing monitoring strategies for deployed machine learning
algorithms: navigating performativity through a causal lens [6.329470650220206]
The aim of this work is to highlight the relatively under-appreciated complexity of designing a monitoring strategy.
We consider an ML-based risk prediction algorithm for predicting unplanned readmissions.
Results from this case study emphasize the seemingly simple (and obvious) fact that not all monitoring systems are created equal.
arXiv Detail & Related papers (2023-11-20T00:15:16Z) - Safety Margins for Reinforcement Learning [53.10194953873209]
We show how to leverage proxy criticality metrics to generate safety margins.
We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
arXiv Detail & Related papers (2023-07-25T16:49:54Z) - Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers.
We then present the pointwise feasibility conditions of the resulting safety controller.
We use these conditions to devise an event-triggered online data collection strategy.
arXiv Detail & Related papers (2022-08-23T05:02:09Z) - Benchmarking Safety Monitors for Image Classifiers with Machine Learning [0.0]
High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation.
The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state.
This paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers.
arXiv Detail & Related papers (2021-10-04T07:52:23Z) - Sample-Efficient Safety Assurances using Conformal Prediction [57.92013073974406]
Early warning systems can provide alerts when an unsafe situation is imminent.
To reliably improve safety, these warning systems should have a provable false negative rate.
We present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics.
arXiv Detail & Related papers (2021-09-28T23:00:30Z) - Learning Uncertainty For Safety-Oriented Semantic Segmentation In
Autonomous Driving [77.39239190539871]
We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving.
We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function.
We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
arXiv Detail & Related papers (2021-05-28T09:23:05Z) - Evaluating the Safety of Deep Reinforcement Learning Models using
Semi-Formal Verification [81.32981236437395]
We present a semi-formal verification approach for decision-making tasks based on interval analysis.
Our method obtains comparable results over standard benchmarks with respect to formal verifiers.
Our approach allows to efficiently evaluate safety properties for decision-making models in practical applications.
arXiv Detail & Related papers (2020-10-19T11:18:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.