Benchmarking Safety Monitors for Image Classifiers with Machine Learning

• URL: http://arxiv.org/abs/2110.01232v1
• Date: Mon, 4 Oct 2021 07:52:23 GMT
• Title: Benchmarking Safety Monitors for Image Classifiers with Machine Learning
• Authors: Raul Sena Ferreira (LAAS), Jean Arlat (LAAS), Jeremie Guiochet (LAAS), H\'el\`ene Waeselynck (LAAS)
• Abstract summary: High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state. This paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. Thus, their deployment in safety-critical applications such as autonomous vehicles is still an open issue. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state despite errors of the ML classifier. As the prediction from the ML is the core information directly impacting safety, many works are focusing on monitoring the ML model itself. Checking the efficiency of such monitors in the context of safety-critical applications is thus a significant challenge. Therefore, this paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers. Furthermore, we propose a framework covering the entire pipeline, from data generation to evaluation. Our approach measures monitor performance with a broader set of metrics than usually proposed in the literature. Moreover, we benchmark three different monitor approaches in 79 benchmark datasets containing five categories of out-of-distribution data for image classifiers: class novelty, noise, anomalies, distributional shifts, and adversarial attacks. Our results indicate that these monitors are no more accurate than a random monitor. We also release the code of all experiments for reproducibility.

Related papers

• SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models [75.67623347512368]
  We propose toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs. Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol. Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs.
  arXiv  Detail & Related papers  (2024-10-24T17:14:40Z)
• Learning Run-time Safety Monitors for Machine Learning Components [8.022333445774382]
  This paper introduces a process for creating safety monitors for machine learning components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output.
  arXiv  Detail & Related papers  (2024-06-23T21:25:06Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens [6.329470650220206]
  The aim of this work is to highlight the relatively under-appreciated complexity of designing a monitoring strategy. We consider an ML-based risk prediction algorithm for predicting unplanned readmissions. Results from this case study emphasize the seemingly simple (and obvious) fact that not all monitoring systems are created equal.
  arXiv  Detail & Related papers  (2023-11-20T00:15:16Z)
• Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers [3.130722489512822]
  SPROUT is a Safety wraPper thROugh ensembles of UncertainTy measures. It suspects misclassifications by computing uncertainty measures on the inputs and outputs of a black-box classifier. The resulting impact on safety is that SPROUT transforms erratic outputs (misclassifications) into data omission failures.
  arXiv  Detail & Related papers  (2023-08-23T11:24:28Z)
• Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection [122.4894940892536]
  We present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level. In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss.
  arXiv  Detail & Related papers  (2022-09-25T04:56:10Z)
• Unifying Evaluation of Machine Learning Safety Monitors [0.0]
  runtime monitors have been developed to detect prediction errors and keep the system in a safe state during operations. This paper introduces three unified safety-oriented metrics, representing the safety benefits of the monitor (Safety Gain) and the remaining safety gaps after using it (Residual Hazard) Three use-cases (classification, drone landing, and autonomous driving) are used to demonstrate how metrics from the literature can be expressed in terms of the proposed metrics.
  arXiv  Detail & Related papers  (2022-08-31T07:17:42Z)
• Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations [50.37808220291108]
  This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior.
  arXiv  Detail & Related papers  (2021-11-18T23:21:00Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)
• Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation [93.83369981759996]
  We propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation. We propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning.
  arXiv  Detail & Related papers  (2020-04-09T14:57:57Z)
• Collaborative Inference for Efficient Remote Monitoring [34.27630312942825]
  A naive approach to resolve this on the model level is to use simpler architectures. We propose an alternative solution by decomposing the predictive model as the sum of a simple function which serves as a local monitoring tool. A sign requirement is imposed on the latter to ensure that the local monitoring function is safe.
  arXiv  Detail & Related papers  (2020-02-12T01:57:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.